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ABSTRACT 


In aviation, spatial awareness and spatial orientation are essential for performing 
the task of recovering from an unusual attitude. Degraded spatial awareness, particularly 
in extreme flight situations, may lead to lower operational effectiveness and to loss of 
equipment and, in manned aviation, loss of life. Therefore, improvements in spatial 
awareness are important in complex 3D environments, including both manned and 
unmanned aviation. 

The main goal of this thesis was to determine whether a new prototype display 
design, called WEBER-Box, is a useful alternative or supplement to traditional flight 
instruments for unmanned aviation. In addition we combined the traditional flight 
instrument as well as the WEBER-Box with a colored-coded indication when the aircraft 
entered an unusual attitude. 

In this experiment, the participants executed typical tasks of a UAV-operator. We 
investigated the influence of the WEBER-Box on UAV operator’s orientation 
performance. The important results can be summarized as follows: 

1. significant improvement in correctly solving the orientation tasks 

2. significant reduction in time to solve orientation tasks 

3. color coded indication of unusual attitude significantly decreased the 

response time and reduced the error 

4. the proposed display design was accepted, interpreted, and used to solve 
3D-orientation tasks efficiently 
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I. INTRODUCTION 


Human performance in spatial orientation tasks requires spatial awareness and the 
skills of an operator to transition from the current spatial attitude into an acceptable 
spatial orientation and position. Degraded spatial awareness (SA), particularly in extreme 
flight situations, may lead to loss of equipment and reduced operational effectiveness. 
Accidents and loss of aircraft due to spatial disorientation (SD) are not uncommon in 
manned aviation. Loss of SA also occurs in unmanned aviation, even with operators on 
the ground. Surprisingly, losses in manned aviation over the last few years are low and 
decreasing but in unmanned aviation, they are high and increasing. 

How to maintain and improve SA is one of the major issues in complex 3D 
environments. In particular, orientation tasks in 3D environments with many degrees of 
freedom are very difficult to accomplish (Previc & Ercoline, 2004). According to Tsang 
& Vidulich (2003) pilots are selected carefully and receive extensive training to maintain 
their spatial awareness, even in extreme fight situations. 

Different classes of Unmanned Aerial Vehicles (UAV) systems operate in a wide 
variety of operational environments, from safe, remote locations to areas under 
immediate threat of enemy fire. This range of operational environments, combined with 
future plans for single-operators controlling multiple UAVs, leads to complex human- 
system interfaces. 

However, the dynamic field of high-flying and Combat UAV demands skilled 
operators who are capable of controlling the aircraft remotely. For large UAVs, pilots are 
usually the first choice for this task. 

The cockpit design of UAVs follows the guidelines of traditional flight 
instrumentation (McCauley & Matsangas, 2004). A legitimate question is whether a new 
prototype display design might improve spatial orientation, especially with an underlying 
color-coding feature when reaching an unusual attitude. McCauley & Matsangas (2004) 
also showed that maintaining spatial awareness and preventing spatial disorientation is a 
key factor when operating a UAV. 
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Since the improvement of the operational utility of UAVs, they have increased 
rapidly in the armed forces of many nations. They have proven to be valuable in recent 
military operations worldwide. However, the rise in UAV use has been accompanied by a 
very high mishap rate due to a number of different causes (Williams, 2004). 

The main motivation for this thesis was to investigate the use of an alternative 
display design for UAV operators for the purpose of improving spatial awareness and 
thereby reducing mishaps. To evaluate this issue, a research questionnaire was sent to 
several US Air Force units and the information derived from that source helped to define 
the research objectives. 

The main goal of the thesis was to determine whether a new prototype display 
design, called WEBER-Box, is a useful alternative or supplement to the traditional flight 
instruments for determining and correcting for unusual attitudes. The application 
specifically investigated was operation of a UAV such as Predator. Another objective 
was to determine whether non-aviator operators will perform as well as aviators operating 
a UAV, even in extreme flight situations or flight maneuvers, when supported with 
different types of attitude indicators. We conducted an experiment where the participants 
performed typical control tasks of a UAV-operator. The experimental task required 
participants to be aware of their spatial orientation and to judge their current attitude 
(pitch and roll orientation). This study investigated the potential of a new display design, 
the WEBER-Box, compared to the traditional flight instruments when performing spatial 
orientation tasks necessary to operate a UAV. The following research questions were 
addressed: 

• Will the new display design help the pilot/operator to recognize the 
current attitude? 

• Will the new display design help the pilot/operator to differentiate 
between a usual or unusual attitude? 

• Will the new design help to prevent misperception of an unusual attitude 
when the UAV is not in an unusual attitude? 

The core research question to be answered was, if the new design supports the 
recognition of the current attitude. An additional research question was, if it aids recovery 
when an unusual attitude exists and help preventing misperception of an unusual attitude 
when the UAV is performing normally. 
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II. BACKGROUND - COGNITION IN AVIATION 


A, SITUATIONAL AWARENESS 

SA is a complex mix of many different factors. In general it can be defined ”as 
the perception, understanding, and the ability to forecast the factors affecting the aircraft 
at any moment in time” (Moroze & Snow, 1999). 

Endsley (1988) provided a first established definition. According to her, SA is 
“the pilot’s internal model of the world around him at any point in time.” It is derived 
from the aircraft instrumentation, the out-the-window view, and his or her senses. 
Individual capabilities, training, experience, objectives, and the ability to respond to task 
workload moderate the quality of the operator’s SA. She expanded in 1995 her definition 
by addressing for the first time three different levels. SA provides “the primary basis for 
subsequent decision making and performance in the operation of complex, dynamic 
systems...” At its lowest level the operator needs to perceive relevant information, next 
he has to integrate the data in conjunction with task goals, and at its highest level, predict 
future events and system states based on this understanding. Vidulich (1995) describes it 
succinctly as referring to the pilot’s cognitive understanding of the current situation and 
its implications. According to Jaslow (1998), SA is the knowledge of the aircraft’s 
location in space (position and attitude), awareness of the environment (terrain geography 
and features, weather-related conditions such as turbulence and icing), understanding 
communication (within the cockpit and via the air traffic controller), correct reading of 
the instruments, and knowledge of malfunctioning controls and/ or instruments. This 
definition provides the wide approach of SA. It also indicates that spatial orientation, as 
shown in a succeeding chapter, is only one part of SA, because it only describes the 
knowledge of the aircraft’s orientation in space (Gawron, 2004). 

Endsley (2000) described this information coherence as an information gap (see 
Eigure 1). 


3 



Data Produced 


Information Needed 



Find 
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More Data ^ More Information 


Figure 1. The information gap (Endsley, 2000) 


In addition, Endsley (1999 and 2000) discusses three different levels of SA: 


Eevel 1: Perception 


Eevel 2: Comprehension 


Eevel 3: Projection 


The first step “is to perceive the status, attributes, and 
dynamics of relevant information in the environment” 
(Endsley, 1999), which is fundamental in a current 
situation. Without the perception of the most 
important information, the option of building an 
incorrect picture of the situation increases. 

Comprehension expands the first level by under¬ 
standing the importance of those elements. 
Combining, interpreting, storing, and retaining these 
information is gaining importance. 

Deriving lessons and consequences from the former 
levels enables the operator to project future events 
and actions. 
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The interaction, not only of the three levels of SA, but also the whole complex of 
SA can be illustrated with Figure 2. 



Figure 2. Model of SA in dynamic decision making (Endsley, 2000) 


B. SPATIAL AWARENESS 

To understand the term “spatial awareness” and to put it into the right context, it 
is important to know that it is only one component of SA. The other components of SA 
are system awareness and task awareness. Each of these components has real-world 
implications, spatial awareness for example for instrument displays. 

Therefore spatial awareness is the understanding of the location, in 3D-space, of 
particular objects within a given environment. Consequently, spatial awareness is 
necessary in order to know, for example, the relative location of enemy targets, friendly 
or neutral forces. Increasing one’s conceptual understanding of a particular 3D-space, 
also increases one’s spatial awareness. It is important to gain a mental picture of the 
current situation because with this ability a pilot or operator is more able to predict the 
future situation and thus be able to operate the aircraft proactively (Wickens, 2002). 
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Wickens (2002b) also describes six different variables that are important for the 
mental picture of a pilot: 

• pitch 

• roll 

• yaw 

• altitude 

• deviation from a flight path 

• position along a flight path 

Thus flight instruments which are displaying information about these variables in 
an appropriate manner contribute significantly to the spatial awareness of a pilot or 
operator of a UAV. 


C. SPATIAL ORIENTATION / DISORIENTATION 

In general, humans are adapted to maintain spatial orientation on the ground. 
Since they are not designed for the three-dimensional environment of flying, which is 
unfamiliar to their body systems, sensory conflicts and illusions are almost inevitable. 
Therefore, spatial orientation is sometimes difficult, even impossible to achieve 
(Antunano, 2003). Three sensory systems (see Figure 3 and Figure 4) provide the 
information necessary to maintain the equilibrium and determine where we are and how 
we are orientated: 

• Visual system: The eyes, which sense position and movement based on 
optical information. 

• Vestibular system: Organs of balance found in the inner ear that sense 
linear and angular acceleration and support posture and locomotion. 

• Somatosensory system: Nerves in the skin, muscles, and joints, 

which, along with hearing, sense position and movement based on gravity, 
feeling, and sound. 
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Figure 3. Spatial orientation (Antunano, 2000) 



Figure 4. Spatial orientation (Antunano, 2003) 

Spatial orientation in flight is diffieult to aehieve beeause the three different 
sensory systems vary in magnitude, direetion, and frequeney. Any differenees or 
diserepaneies result in a sensory mismateh that ean produee illusions and lead to spatial 
disorientation. Good spatial orientation relies on the effective perception, integration and 
interpretation of their sensory information (Antunano, 2003). Thus one major issue is the 
question of appropriate countermeasures. The most common one is to “trust your 
instruments.” On the other hand, flying by instruments increases demands on mental 
resources, task load, and disorientation stress (Braithwaite, Dumford, DeRoche, Alvarez, 
Jones, Hidgon et ah, 1997). 
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SD is a failure “to sense correctly the position, motion, or attitude of the aircraft 
or of him/ herself within the fixed coordinate system provided by the surface of the earth 
and the gravitational vertical” (Benson, 1988). In addition, errors in perception by the 
aviator of his position, motion, or attitude with respect to his aircraft, or of his own 
aircraft relative to other aircraft, may also be embraced within a broader definition of SD. 
Any condition that deprives the pilot of natural, visual references to maintain orientation 
can rapidly cause SD. Regardless of a pilot’s experience or proficiency, sensory illusions 
can lead to differences between instrument indications and what the pilot feels the aircraft 
is doing. 

SD in flight operations has been a common problem since the early days of 
aviation. Accidents continue to occur despite improvements in the display of information 
to the pilot. Even with improvements of manufacturing technologies, quality control, and 
aircraft maintenance, accidents still occur. Reasons for this fact are related for example to 
new technologies such as night vision goggles. Flight operations can now be executed in 
environments that have not been possible before (Benson, 1988). 

Between 1990 and 1999, the United States Air Force (USAF) experienced 36 SD- 
related Class A mishaps. These have cost a total of $557M and even more important, the 
loss of 44 aircrew. These economic consequences of SD are enormous, both in cost of 
lost aircraft, lost aircrew, and cost of training new aircrew (Heinle & Ercoline, 2003). For 
example, in the period 1992 - 2000, SD was the cause of 20.2% of USAF Class A 
mishaps. Respectively US Army with 27% and US Navy with 26% rate in an equivalent 
period. But on the other hand, in a period of 1972 - 2000 USAF Class A mishaps have 
decreased in an overall rate from 2.5 accidents per 100.000 flying hours in 1972 to a rate 
of 1.0 in 2002. Despite this decrease, the rate of SD has only fallen from 0.5 to 0.25 and 
has remained constant in the last 15 years (Benson, 2000). Therefore, in general, SD is 
still the single most common cause of human-related aircraft accidents (Heinle & 
Ercoline, 2003). 

The literature distinguishes between three different types of SD. About 80% of all 
Class A SD accidents are of type I and unrecognized. Pilots do not consciously perceive 
any manifestation of SD and most often it occurs when he/ she breaks a cross-check. 
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Most likely it also leads into a controlled flight into terrain. 20% of all SD Class A 
accidents are form type II. The pilot consciously perceives a manifestation of SD but may 
not attribute it to SD. Often, he suspects an instrument malfunction. The most well- 
known example is the Graveyard spin (see Figure 5). In this case the pilot realizes a 
conflict between the information displayed by the instruments and the feeling generated 
by the inner ear. The pilot has to decide which sensory system to believe. If he/ she trust 
his/ her inner ear, the aircraft may spin all the way to ground impact. The last type, type 
III, of SD, also called incapacitating SD, is the least common, known and understood. 
Few written reports exist at all, but it is known through experience and pilot reports. An 
example is named “giant hand illusion” (see Figure 6). Sometimes pilots feel that they are 
unable to move the controls in one direction, although they are capable of moving the 
controls in any other direction. A suggested countermeasure is to remove the hands from 
the controls and try it again by only using fingers without arm movements. Even a 
technical inspection after the flight showed no evidence of a technical malfunction and it 
is not possible to duplicate the problem (Heinle & Ercoline, 2002). 



Eigure 5. Graveyard Spin (Heinle & Ercoline, 2002) 
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Figure 6. Giant Hand Illusion (Heinle & Ercoline, 2002) 


Each type of SD affects the pilot in a different way, and each should be 
thoroughly understood by the pilot before he or she experiences them in flight. Although 
the phenomenon of SD has been described and documented by many, both researcher and 
aircrew, since the earliest days of aviation, a complete understanding of the complex 
mechanisms and interactions has remained elusive. 

According to Benson, 2002, the human factors and aero-medical community also 
has to be aware of SD in virtual reality and in operating UAVs. He defines SD as a 
misperception of the key elements in the flight environment. These key elements may 
include navigation, weather, tactics, nature of threats, aircraft systems, and spatial 
orientation. The task of piloting a UAV may be sensitive to SD, especially in cases of 
highly maneuverable vehicles, such as Combat-UAV where information is provided by 
optical and other sensors, and in extreme environmental conditions, like high wind 
speeds. In these cases, he concludes, there is a considerable potential for pilot 
disorientation and loss of control. Several factors, like restricted field of view, and lags in 
the display of information can be causes of mishaps. In addition, there is also the lack of 
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motion stimuli to the operator’s body, in contrast to manned aviation. Unlike in 
conventional aviation, SD is widely underestimated for unmanned aviation. Since 
unmanned aviation lacks the direct physical perception and feedback, we expect that 
improved display concepts will contribute to a better spatial awareness and, hence, will 
decrease the potential for SD. A 10-year cross sectional review of human factors in UAV 
mishaps within the U. S. Department of Defense (DoD) supports Benson’s position. 
Interestingly, the outcome of this DoD-review is that the misperception error was present 
in 5% of all UAV mishaps. 
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III. COCKPIT AND DISPLAY DESIGN 


A, GENERAL PRINCIPLES DISPLAY DESIGN 

The design of flight instruments and the cockpit panel have a long history, 
beginning with Elmer Sperry and the Airborne Gyro (Hughes, 1971). Even in the earliest 
powered flights, keeping the simple planes steady was difficult. The addition of 
gyroscopes in 1914 brought new instruments to the cockpit, helping the pilot to maintain 
steady control. Gyroscopes and their derivatives remain an important part of flight 
technology. Basically they have not changed since the beginning of aviation. Moreover, 
they have been, and still are, the pilot’s main source of information. They are his tool for 
flying in an usual attitude, recovering from an unusual attitude, and for preventing 
mishaps. 

Traditional cockpit design and flight instruments have almost always been arrayed 
with the most important instruments in the shape of a T (see Eigure 7). This configuration 
can be found in a wide variety of aircraft from the beginning of aviation to the latest 
models of aircraft. 





Eigure 7. Basic layout of flight instruments (Anderson, 2002) 
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The center spot is always reserved for the artificial horizon or attitude indicator, 
which is not only one of the oldest instruments but also one of the most important. It 
provides information to the pilot about his current situation in space and therefore 
displays the flight attitude relative to the earth (see Figure 8). Pitch attitude is indicated 
by up and down motion of the sphere with respect to horizontal reference. Roll attitude is 
indicated by rotational motion of roll pointer with respect to the fixed roll scale located at 
top of the display. 



Figure 8. Traditional attitude indicator (screenshot of 
X-Plane ™ flight simulation) 

The evolution of aviation from the beginning with the Wright brothers’ to modern 
aviation has brought some significant changes in the capabilities of the aircraft (Tsang & 
Vidulich, 2003). Besides the instruments, which basically remained the same, other 
factors like speed of the aircraft, especially for fighter aircraft, have changed. The 
demands on the pilots have thus increased, the workload is higher, and the task is more 
complex now. 

With the development of new technologies in flight instrument design, there has 
been a shift away from individual displays like altimeter, airspeed indicator, or tum-and- 
bank indicator, towards integrated displays in which most of the information comes from 
a common source like a heads-up display (HUD) or helmet-mounted display (HMD). 
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Independent of the state-of-the-art display design, three basic issues can be 
“applied to the design and modification of primary flight and navigation instruments in 
the context of situation awareness assessment” (Andre, Wickens & Moorman, 1991): 

• Display perspective: two- vs. three-dimensional 

• Frame of reference: inside-out vs. outside-in 

• Visual display monochrome vs. color 


Many displays, like traffic, weather, or navigational displays, are two-dimensional 
with a “God’s eye” plan view of the earth. On the other hand, flying an aircraft is much 
more related to a three-dimensional task. In this context, three-dimensional means the use 
of “perceptual depth or distance cues to create a three-dimensional image (Wickens, 
2003).” 


An important feature of 3-D displays is the frame of reference or the viewpoint 
from which a selected part of interest is presented (Wickens, 2003). Thus the point of 
view is important for a comparison between flight instruments. Previc & Ercoline, 1999, 
describe the fundamental differences between the first two types of attitude presentation 
for an aircraft in the following way: 

• Egocentric or Immersed view point: This type is also known as “inside- 
out” or “moving horizon” or “moving outside world”. The outside world 
is presented from the inside perspective of an aircraft to the outside world 
(see Figure 9, right). The human is fixed in space and the world around 
him moves. 

• Exocentric or tethered view point: This type is also known as “outside- in” 
or “moving aircraft”. The perspective is from the outside world into the 
aircraft. This type of attitude presentation is rarely used for UAVs (see 
Figure 9, left). The world is fixed and the human moves. 

• Coplaner view point: It shows a top-down lateral view and a vertical 
profile view, which can also have a side view or a back view (Wickens, 
2003). Therefore it is a 2-D display, contrary to egocentric or exocentric 
that are 3-D display designs. 
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Figure 9. Fundamental differences of attitude representation 
for an aircraft (Previc & Ercoline, 1999) 


According to Clay, 1993, many aircraft accidents are due to SD problems of the 
pilot and to an additional wrong interpretation of the flight instruments. One of the major 
challenges for flight instrument display is the tremendous data flow and information 
available from all of the instruments. Additionally, the priority of recalling the 
information changes with every task. In addition, flight instruments are normally not very 
intuitive (Previc & Ercoline, 2004). On the other side, multifunctional instruments and 
the “glass cockpit” have been developed. This approach may lead to an overload of the 
pilot/ operator because more information is available, the number and complexity of 
display pages has increased. Furthermore, with an increasing number of warning signals 
like audio or visual, status displays, flight path controls, only to mention a few, the 
human processing ability may be exceeded (Mejdal & McCauley, 2001). As a result, the 
requirements for developing new display designs or improving existing display designs 
become important. This display design challenge is taken in a new direction when UAVs 
are taken into account. For this purpose, display designs have to be less abstract and 
intuitively understandable for an operator on the ground. 
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The visual impact of color is strong and has been used in simple display 
applications since before the Second World War. The advantage of color for flight 
instruments is enhancing the presentation of information and gaining user acceptance. 
Color has proven effective when a lot of information must be presented. Two different 
applications of color-coding are widely used. First, color can be used to signify quantity, 
that is, to categorize data. The airspeed indicator is an example for this type of 
application. Secondly “qualitative color displays classify data by type or condition and 
are often used to convey status information” (Stokes & Wickens, 1988). For this 
application, color can be used to facilitate selective attention. It can act as an attention 
cue for the pilot who will never be able to monitor all his displays at the same time with 
the same level of attention. Typically colors for danger, caution, or advisory information, 
are red, amber, or green, respectively (Stokes & Wickens, 1988). 


B, INSIDE-OUT VERSUS OUTSIDE-IN DISPLAY DESIGN 

There is a general discussion between researchers as to which type of design of 
attitude presentation human beings prefer. One very early approach to utilizing an 
exocentric instrument design was a study by Hennessy, Lintern & Collyer, 1981, that 
used an exocentric view on an aircraft model to facilitate the flight training of pilots. Four 
different visual displays were evaluated for their effectiveness in the acquisition of flight 
tasks in a flight simulator. The experimental displays were, 

(a) wide-field-of-view (160°) with horizon and checkerboard ground plane 

(b) narrow-field-of-view (48°) with horizon and checkerboard ground plane 

(c) an outside-in view of an airplane, also referred by the term 
“unconventional visual display” 

(d) a display that consisted only of traditional flight instruments. 

The participants were flight students who had to learn basic flight tasks in a 
simulator. The experiment was conducted in form of 20 trials (4 per display type). The 
researchers were particularly interested in the transfer of training in terms of how 
accurate a flight student could perform certain simple flight maneuvers. The key finding 
was that the students were able to learn flight maneuvers faster when using the outside-in 
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display. Furthermore, the results showed that it is advantageous to have an external point 
of view for perceiving and understanding flight behaviors. 

Previc & Ercoline (1999) presented additional findings on outside-in flight 
displays, claiming that they are superior at preventing roll-reversal errors during normal 
flight and in recovering from extreme flight situations. They state that “Although outside- 
in attitude displays have been, and continue to be, flown throughout the world, the 
majority of military and civilian aircraft have failed to adopt this format” (Previc & 
Ercoline, 1999). One of their conclusions is that even experienced pilots do not show a 
clear advantage with an inside-out attitude display with which they have trained 
extensively. In contrast, pilots trained on an outside-in format show overwhelming 
advantage in recovering from unusual attitude. 

Self, Breun, Eeldt, Perry & Ercoline, 2002, conducted a study for students in an 
introductory flight course at the U.S. Air Eorce Academy. They were tested on three 
different display symbologies: 

(a) moving horizon symbology (egocentric viewpoint) 

(b) moving plane symbology (exocentric viewpoint) 

(c) arc-segment attitude reference symbology. 

The researchers evaluated the time a student needs to give a roll-input in order to 
correct a certain spatial orientation of the aircraft as their main parameter of interest. This 
study found that the exocentric “outside-in” attitude display and the arc-segment attitude 
reference symbology were superior in reducing the time of roll-inputs in normal flights. 

Tsang & Vidulich, 2003b, confirmed the former findings that pilots prefer an 
exocentric or outside-in view of the world. 

C. PICTORIAL DISPLAY DESIGN 

Research on pictorial display design began in early 1990s when National 
Aeronautics and Space Administration (NASA) conducted a series of experiments in 
order to determine if landing and taxing approaches of commercial aircraft and Space 
Shuttles may benefit from an external view of the own aircraft. Additionally Parrish, 
Busquets, Williams & Nold, (1994) conducted an experiment to evaluate certain display 
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types, including a pictorial display design, which is also called “pathway in the sky” or 
“highway in the sky”. The objective was to evaluate and compare spatial awareness 
components of pilots using various display concepts. These four are: 

(a) conventional electronic flight information systems (EFIS) 

(b) EFIS and speech commands 

(c) pictorial “pathway in the sky” display with 70° field-of view and 

(d) a pictorial “pathway in the sky” display with 40° field-of view. 

The pictorial display was a computer rendered graphic of an out-the-window 
scene with a flight path that was displayed as green goalposts with width and height in 
correspondence with the input from the electronic landing system. The results showed 
greatly improved subjective rankings of the pictorial design in terms of reduced 
workload, effort to monitor traffic and overall ranking. The objective measurements 
reveal a significant decrease of maneuver errors in all monitored categories. 

This result is also supported by Ercoline, DeVilbiss & Evans (2004) because the 
pictorial design benefits the pilot’s spatial orientation. The configuration of the display 
appears more like the real world because a symbolic path is used to represent “the desired 
vertical and horizontal trajectory relative to Earth’s surface as opposed to command 
steering bars or a single command steering cue (Ercoline et ah, 2004).” In addition, the 
pictorial display is appreciated by novices and experienced pilots. 


D, THE WEBER-BOX 

The WEBER-Box is a design concept for flight instruments that emphasizes a 
three-dimensional exocentric view of a virtual aircraft (Weber, 2006). He proposed in his 
thesis a new human-centric design approach for a 3D-flight instrument based on “avatar” 
concepts. Weber (2006) combined an exocentric design with pictorial metaphors of the 
current spatial orientation. By animating the entire scene, a story is being told about what 
happens during a flight maneuver. In this context, it is a “miniature abstract virtual 
world” that represents the actual spatial situation of the aircraft. The avatar “lives” in this 
virtual world. The display representing the aircraft is a 3D wire-frame model that hovers 
inside an abstract box and replicates all the basic motions of the real aircraft. The frame 
of reference is a coordinate system, consisting of X-, Y-, and Z-axes. Altitude is 
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represented by a 3D bar along the Y- axis. The aircraft’s movement relative to the ground 
is displayed by moving objects along the X-/Z- plane (see Figure 10 and 11). 

Weber (2006) conducted two experiments, which found strong evidence for 
improvement of spatial awareness and pilot’s performance in extreme flight situations. 
His findings were that the participants were able to judge the spatial orientation of their 
own aircraft about three times faster, compared to traditional flight instruments. The time 
needed to recover the aircraft from unusual attitudes improved by about 70%. 
Furthermore, he found no statistically significant differences in terms of performance 
improvement between pilots and non-pilots. Thus, operators of any experience level 
might benefit from the design of the “WEBER-Box”. 





Eigure 10. Screen shot of the WEBER-Box 
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Figure 11. Screen shot of the WEBER-Box 


Studies show that maintaining spatial awareness and preventing SD is becoming 
more and more an issue when operating UAVs. Current UAV controls rely on traditional 
flight instruments, which are inherently difficult to understand and demand a high degree 
of training (McCauley & Matsangas, 2004). On the other hand, human-centric display 
design is likely to substitute or supplement traditional flight instruments in the future. The 
outside-in display formats, combined with pictorial visualization seems to be promising 
to support and improve spatial awareness for operators of UAVs. The design concept of 
the WEBER-Box combines those display principles. 
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IV. UNMANNED AERIAL VEHICLES 


A, HISTORY AND OVERVIEW 

Traditional aircraft have an important disadvantage or weakness. They have to be 
piloted by humans. Therefore there are negative consequences which have to be 
considered. In case of a crash, you not only loose equipment worth millions of dollars but 
also may loose the pilot. Other negative outcomes in military operations include the pilot 
and aircrew being taken as prisoners of war or hostages, which could have a serious 
impact on public opinion or the way a conflict has to be fought. 

Thus, the idea of developing uninhabited or unmanned aerial vehicles arose. Since 
1964 the United States Military has spent billions of dollars researching, developing and 
constructing countless types of UAVs. The United States Air Force has used UAVs for 
reconnaissance and surveillance at very long ranges up to 3,000 miles away. The core 
idea is to provide the warfighter with intelligence information and a precise delivery 
mechanism for smart area weapons, non-lethal payloads, as well as other cargo. Recently 
UAVs have also been established by the US Army and Marine Corps as tactical assets on 
the battlefield. Figure 12 provides a good impression of the actual size of a UAV Predator 
and Global Hawk compared to a Boeing 737 and therefore how far the development for 
large UAVs has already gone. 



Figure 12. Sizes of UAV Predator and Global Hawk compared to a Boeing 737 

(Picture by Northrop Grumman) 
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On the other hand, the latest and additional development points into the direction 
of Micro UAVs which can be launched, for example, in urban warfare. 


UAV Evolution - Where are we? 
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Figure 13. UAV Evolution - Where are we? (AUVSI, 2001) 


There is not only the question “Where are we?” but even more interesting is the 
question “Where are we going?” Currently under development is the Automated Aerial 
Refueling (AAR) that could demonstrate for the first time a UAV’s ability to 
autonomously maintain a permanent refueling station behind a tanker aircraft. This 
extends the range of a UAV tremendously because now only the crew in the GGS has to 
be exchanged but the UAV can continue its mission. 

Even the goals for developing and building future UAVs are determined, there are 
still concerns (see Table 1) that have to be taken care of (Thompson, 2000). 
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Datalink 

Communica¬ 

tions 

1. Loss of control due to enemy jamming or signal manipulation 

2. Long connectivity lapses due to distance, satellite location, or friendly mutual interference 

3. Limited amount of frequency bandwidths to accommodate large numbers of secure links 
for muttiple UCAV operations 

Air Refueling 

1. Transoceanic deployment distances and communications 

2. Risk to KC-135 or KC-10 high-value assets 

3. Tanker joinup and multiaircraft air refueling 

Operator 

Situational 

Awareness 

1. Number of aircraft per operator or operators per aircraft 

2. Air traffic control (ATC) and enemy airspace deconfliction from other aircraft 

3. Threat reactions for visual antiairaaft artillery, infrared surface-to-air missiles, or enemy 
aircraft 

Emergencies 

1. Less capability to rapidly assess and correct aircraft problems 

2. Unable to see damage, feel small vibrations, or smell smoke 

3. UAV-capable alternate airfield recovery due to fuel or weather 


Table 1. Future UCAV eoneems (Thompson, 2000) 


B. ACCIDENTS 

High mishap rates are often cited as a deterrent to a wider adoption of UAVs into 
future force structure. Over the last five decades, investments in manned aircraft 
reliability have been made to drive equipment failures to near zero. This progress implies 
that a considerable reduction in UAV accident rates due to technical failure also can be 
obtained with reasonable investments. On the other hand, it is also likely that the human 
in the loop contributes as a main source of failure and high mishap rates for both manned 
and unmanned aviation. A significant contributor to UAV mishaps is the experience level 
of UAV operators and maintainers. 

Williams (2004) classified UAV accident data into the categories of human 
factors, aircraft, and unknown. He analyzed explicitly the data of the following UAVs: 
Hunter, Shadow, Pioneer, Predator, and Global Hawk, which are currently in use in the 
American Forces. His findings are that the involvement of human factors was high, in a 
range from 21% to 68%, depending on the type of UAV. Various human factors issues 
have been observed, including handing over a UAV from one pilot to another, launch and 
recovery, display issues, and disorientation. 

DoD (2005) compared in its roadmap 2005 - 2030 the UAV mishap rate with the 
mishap rate of the F-16 (see Figure 14) of class A mishaps. These are the mishaps 
resulting in the loss of the aircraft, human life, or causing $1,000,000 in damage. It 
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illustrates the mishap rate after similar numbers of flying hours have been accumulated. 
This Figure shows significantly that the mishap rate of today’s UAV is comparable with 
the mishap rate of early manned military aircraft. 



Cumulative Flight Hours 


Figure 14. Mishap rate comparison (DoD, 2005) 

An additional approach is the vulnerability of UAVs in combat situations. Many 
of the current UAVs are vulnerable to different air defense systems, which of course 
depends on the altitude and the speed of the UAV. Certainly this applies to tactical UAVs 
up to Medium-Altitude, Long-Endurance (MALE) UAVs like the Predator. Anti-aircraft 
artillery (AAA), shoulder-fired man portable systems (MANPADS), and radar directed 
low medium and high altitude surface to air missiles can all be quite lethal to UAVs 
within their range. Currently the concept of operations calls for air superiority and the 
elimination of the air defense threats before a UAV is sent into this area. In the future, the 
concept of operations could be extended if active countermeasures were installed on 
UAVs, which would enable their mission to be extended to surveillance prior to a 
conflict. 
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C. UAV SYSTEM EXAMPLE: PREDATOR 

Smaller in size eompared to its larger brother Global Hawk, the 27-foot-long 
UAV Predator is mueh eheaper, $4.5 million eaeh. As a result, it may be worth the risk of 
being sighted and shot-down to fly a Predator at relatively low altitudes (25,000 feet and 
below). 

Predators were first deployed for reeonnaissanee and surveillance operations by 
the US military during the 1995 civil war in Bosnia. They are now far more extensively 
used by the USAF in the operations such as Afghanistan. 

The Predator is a medium-altitude, long-endurance remotely piloted aircraft. Its 
primary mission is interdiction and conducting armed reconnaissance against critical, 
perishable targets. When it is not actively pursuing its primary mission, it acts as the Joint 
Forces Air Component Commander-owned theater asset for reconnaissance, surveillance 
and target acquisition in support of the Joint Forces Commander. 

The basic crew is one air vehicle operator (AVO) or pilot and two sensor 
operators. They fly the aircraft from inside the ground control station via a line-of-sight 
data link or a satellite data link for beyond line-of-sight flight (see Figure 15). 


Ku-Band 




- 


C-Band LOS 

Predator 


Primary 
SATCOM Link 


1 •• 

Ground Control Station 



Figure 15. Data and satellite link of the Predator (Cooter, 2001) 
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The Predator is equipped with a color camera in the nose (generally used by the 
pilot for flight control), a day variable-aperture TV camera, a variable-aperture infrared 
(IR) camera (for low light/night), and a synthetic aperture radar (SAR) for looking 
through smoke, clouds or haze. The cameras produce full motion video while the SAR 
produces still-frame radar images. Thus, it can produce near-real-time multi-sensor 
imagery and intelligence. The latest development of the RQ/MQ-1 is equipped with 
lasers, targeting systems, and a pair of hellfire missiles. The following model “MQ-9 is 
intended to fly higher and faster, provide more power, and carry larger payloads than the 
original Predator system. It is intended to provide a more robust airframe, using a 
conventional turbo-prop engine and redundant avionics. MQ-9 will be used primarily in 
an armed reconnaissance (“hunter-killer”) role and will perform reconnaissance, 
surveillance, and target acquisition (RSTA) as a secondary role” (Christie, 2003). 

One has to understand the Predator as a complex system which includes 
• 4 X Air Vehicles 



Figure 16. UAV Predator (Cooter, 2001) 
• lx Ground Control Station 



Figure 17. Predator Ground Control Station - outside (Cooter, 2001) 
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Figure 18. Predator Ground Control Station - inside (Cooter, 2001) 
• lx Predator Primary SATCOM Link 



Figure 19. Predator Primary SATCOM Link (Cooter, 2001) 


For operating the Predator a HUD is being used. Figure 20 shows the current one 
which will be replaced in the future by a more advanced IHUD (see Figure 21). 
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Figure 20. Heads-Up-Display Predator (Ball et al., 2002) 



Figure 21. Heads-Up-Display Predator - IHUD (Guy, 2007c) 


The Predator HUD can be considered as one of the traditional flight instruments 
which we want to compare with the new prototype display design in the present research. 
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Next to the HUD/ IHUD the pilot uses the front color camera for operating the 
Predator (see Figure 22 and Figure 23). 



Figure 22. View from the color camera in the nose 
of the Predator (Guy, 2007b) 



Figure 23. View from the color camera in the nose of the 
Predator during recovery (Guy, 2007b) 
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D, GROUND CONTROL STATION 

The ground control station (GCS) is the control center of any UAV system. The 
main tasks that have to be accomplished are data processing of the incoming data and 
sending control signals to the UAV. Thus three main features will be executed in the 
GCS: 

• Mission planning: The purpose of mission planning is to analyze available 
information and develop a flight plan. This involves identifying and 
prioritizing targets, and examining sensor capabilities relative to the 
chosen targets. Weather analysis also is common. 

• Mission control: Mission control is responsible when the aircraft is on the 
ground, when it is getting launched, and during the mission, for the 
navigation and possible mid-flight rerouting. It is also responsible for 
monitoring and controlling of any payload, and for recovering the aircraft 
if an unusual attitude occurs. 

• Data manipulation: Data manipulation is responsible for processing, 
exploiting, and archiving of data during the flight and / or after the flight 
(Anderson, 2002). 


The Predator GCS provides command and control of the air vehicle through the 
pilot station using stick and rudder control. The sensor operators monitor the flow of 
information from the UAV. Their sources of information are the TV camera, the infrared 
camera, and the onboard radar. Depending on the weather conditions, advantageously the 
radar can be operated simultaneously with either the TV camera or the infrared camera. 
For communication purposes, a C-band line-of-sight data link and for beyond-line-of- 
sight a Ku-band satellite data link are used (Anderson, 2002). 

The following pictures give an idea about the inside of a GCS. For comparison 
reasons, not only the GCS of the Predator will be displayed, but also several others. 
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Figure 24. GCS Predator - Inside view (Cooler, 2006) 



Figure 25. GCS Predator - Inside view 
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Figure 26. GCS Predator - Inside view (Cooler, 2006) 



Figure 27. GCS Pioneer - Inside view 
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Figure 28. GCS Hunter - Inside view 



Figure 29. GCS of German System KZO - Inside view 
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V. METHOD 


A, METHODOLOGY 

1. Experiment Goals and Purpose 

The goals of this study are to evaluate the strengths and weaknesses of a new 
proposed display design, called the WEBER-Box, compared to the traditional flight 
instruments. 


2, Research Questions 

The study is limited in its scope and assets. We wanted to answer the following 
questions: 

a) Will the new display design help the pilot to recognize the current aircraft 
attitude? 

b) Will the new display design help the pilot to differentiate between a usual 
and unusual attitude? 

c) Will the new design help to prevent misperception of an unusual attitude 
when the UAV is not in an unusual attitude? 


Answers to these questions reveal whether the new display design helps to 

• recognize the current attitude at least as well as the traditional display 
design, 

• differentiate between a usual and an unusual attitude at least as well as the 
traditional design, and 

• determine whether the new design prevents misjudging an unusual 
attitude. 

3. Constraints and Assumptions 

Since the experiment involved human participants, the plan for the experiment 
was submitted to the Institutional Review Board, which approved the experiment (see 
Appendix C). 
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The major constraint was the use of a digital hand stopwatch, type SPORTLINE®. 
Therefore the accuracy of measuring the time to judge the current attitude may have been 
decreased, but this possible disadvantage was constant across conditions and, presumably 
across data points (1,120). 

The major assumption was about the definition of unusual attitude. According to 
Air Force Instruction II-2I7 “an unusual attitude is an aircraft attitude occurring 
inadvertently”. Since this is very broad, we followed a definition according to industry 
...”as one where pitch unintentionally exceeds 25 degrees nose up or 10 degrees nose 
down, bank angle unintentionally exceeds 45 degrees, and the airspeed is inappropriate 
for the conditions” (Guy, 2007a). 

4. Technical Equipment 

The experiment took place at the Naval Postgraduate School, Monterey, 
California, Watkins Building, room 212B. 

Two workstations were used. Workstation one was for instruction and training 
while workstation two was established for the underlying experiment. 

Workstation one: 

The computer-generated images were displayed on a 19” computer monitor with a 
1024 X 768 pixel resolution. The pictures of the flight situations were generated by 
screenshots of the commercial flight simulator software X-Plane'^’^, version 7.13 and 
version 8.40 and by screenshots of the WEBER-Box program, respectively. The pictures 
were stored in a PowerPoinEM slideshow (see Appendix D) to ensure that every 
participant saw the different attitudes in the same order. 

Workstation two; 

The computer-generated images were displayed on a 19” computer monitors with 
a 1024 X 768 pixel resolution. The second screen was located on top of each other similar 
to the ground station of a Predator. It provided during the duration of the experiment the 
definition of unusual attitude. 
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The simulation of the flight situations were generated by the commercial flight 
simulator software X-Plane™, version 7.13 and version 8.40 and by the WEBER-Box 
program, respectively. The predefined simulations of usual and unusual attitude will be 
stored in Camtasia Studio'^’^ from TechSmith to ensure that every participant saw exactly 
the same the different attitudes in his randomly assigned order. 

5. Data Collection Methodology 

Several types of data were collected during the experiment. 

Eirst, demographic data were collected by a pre-experiment questionnaire. 
Secondly data were collected during the experiment. These were data for correct/ 
incorrect responses, and the measured time each participant needed to judge the current 
attitude. In addition, the use of subjective self-assessment provided a scale of five 
judgments in every questionnaire. 

In summary, a “repeated measures” design was used in which all the participants 
were exposed to all the instrument setups and all the sub-experiments. The order of 
presentation of the two displays and the color-coding was pseudo-randomized (see 
Appendix J and K). This procedure ensured that learning effects would be 
counterbalanced over the experiment. 

6. Data Analysis Methodology 

The data was hand recorded by using a template (see Appendix K). The 
participant judged the current attitude and responded by saying “Usual attitude” or 
“Unusual attitude”. The experimenter measured the time and transferred the time into the 
template together with the participants’ verbal response of his judgment. 

The data then were analyzed by commercial software Excel 2003 and SPSS 13.0 
for Windows. Methods that have been used were Analysis of Variance and the paired t- 
test. The statistical analysis concentrated on the differences between the measurements 
depending on the experimental condition: display type (traditional or WEBER-Box) and 
color coding (none or yellow). 
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Two questionnaires were handed out. The pre-experiment questionnaire provided 
demographic information and, for pilots, aviation background. The post-experiment 
questionnaire was helpful as a secondary method for determining the effects of the 
different variables display design and color/ no color, and to analyze the relationship 
between subjective and quantitative data. 


B, EXPERIMENTAL DESIGN 

1. Basic Experimental Design 

The experiment was designed to compare the correct responses of the judgment of 
usual or unusual attitude while changing the variables display design and color coding or 
not. Therefore, it was a randomized “within subjects” design because every participant 
received the same treatment. Since two groups, pilots and non-pilots, were involved in 
the experiment, their comparison was “between subjects.” 

The dependent variables were the outcome of correct response and the time a 
participant needed to judge the attitude. Independent variables were the display design 
and the factor color or no color. 

Each participant was told at the beginning that his duty station is the ground 
control station of a Predator. 

2. Experimental Procedure 

The overall experiment was subdivided into four phases. 

Phase one - Pre-Paperwork: 

The participant filled out the necessary paperwork, including a consent form, a 
privacy act statement, a minimal-risk information form, and an initial/ demographic 
questionnaire. 
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Phase two - Introduce Training 

Then the participant was introduced to the experiment. In a PowerPoint 
presentation he received information about the UAV, his task as an operator of the UAV, 
and details about usual and unusual attitude since this was the key part of the underlying 
experiment. The participant also watched a 2-minute video of the work as an operator in 
the ground control station of the Predator. 

This pre-experimental part, phases one and two, took approximately 25 minutes. 

Phase three - Experiment 

After completion of introduction and training, the participant started with the first 
experimental sub-task, given a particular simulation setup, that is, with or without the 
new instrument. Each of the 14 trials of the sub-experiment was scheduled for 10 
seconds. Two monitors were placed on top of each other to create an environment like a 
ground station of a UAV. The top monitor was only used for displaying the definition of 
unsual attitude. The first two trials in each combination always were practice trials. 

The experimenter started the prepared simulation of 19 different flight simulations 
that ended in a usual or unusual attitude. Each simulation appeared for ten seconds. This 
was also the amount of time given to the participant to judge the attitude. After ten 
seconds, the next attitude appeared automatically. One half of the trails were created as 
usual attitude while the other half as unusual attitude. 

After accomplishing the first two blocks of trials, the participant was allowed a 5- 
minute break if desired. 

Then the participant started the second sub-experimental task. 
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Intro and 
Initial 
Briefings 


Break 


Debriefing 



Figure 30. Experimental setup 


In summary, all the participants were exposed to all the instrument setups and all 
the sub-experiments. Each participant was confronted with seven trails of usual attitude 
with traditional design and seven trails of unusual attitude with the traditional design and, 
likewise with the new design (see Eigure 30). 

Phase four - Ppst^Paperwork 

When finished, the participant was asked to fill out the post-experimental 
questionnaire and support the experiment with his suggestions and comments. This 
completed the experimental session. 


C. PILOT STUDY ONE 

The first pilot study was conducted with three participants to test the procedures, 
apparatus, and tasks. The setup of this pilot study differed from the later conducted 
experiment. It was a dynamic environment with a flight simulation that stopped after 120 
seconds and the participant had to judge the current attitude. Eor this judgment he was 
given ten seconds. There was also only the comparison between the traditional display 
design without color coding and the new display design with color coding. Two of the 
three participants claimed that this was a disadvantageous setup due to the outstanding 
support the new display design received. The recommendations were to extend the setup 
and provide both of the display design types with color and compare the results. 
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D, PILOT STUDY TWO 


Therefore another pilot study had to be conducted according to the above 
mentioned recommendations. Again three participants tested the procedures, apparatus, 
and tasks. The new setup contained now both display designs, each with and without 
color coding. We also changed the settings contrary to the former pilot study to a static 
environment with a presentation mode where the participant had to judge the attitude in a 
maximum time of ten seconds. If he was unable to judge in this period, it was counted as 
incorrect. All of the participants were tested on each of the four combinations. The pilot 
study demonstrated that every participant was able to judge the current attitude and the 
given time of ten seconds sufficed. 


E, PARTICIPANTS 

The participants were randomly assigned on a volunteer basis from students and 
faculty of NPS. The study involved 20 participants, evenly partitioned into ten pilots and 
ten non-pilots. Eight of the pilots were jet pilots and two were helicopter pilots with an 
average of 1,348 flight hours. Therefore the group of pilots can be considered as 
participants with a strong aviation background and a high flight experience. The mean 
age of the pilots was 36 in the range from 28 to 42. The mean age of the non-pilots was 
34 with a range from 28 to 42. 
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VI. DATA ANALYSIS 


A. HYPOTHESIS 

Inferential testing techniques enable the researcher to decide between the null 
hypothesis and the alternative hypothesis. Devore, 2004, defines the null hypothesis, 
“denoted by a, as the claim that is initially assumed to be true.” The alternative or 
secondary hypothesis, “denoted by , is the assertion that is contradictory to . The 
null hypothesis will be rejected in favour of the alternative hypothesis only if sample 
evidence suggests that //q is false. If the sample does not strongly contradict we will 
continue to accept the null hypothesis. The two possible conclusions from a hypothesis¬ 
testing analysis are then reject or fail to reject Hf' (Devore, 2004). 

A test procedure is specified by the following: 

• A test statistic, a function of the sample data on which the decision (reject 
Hq or do not reject ) is to be based. 

• A rejection region, the set of all statistic values for which //y will be 
rejected. 

The null hypothesis will then be rejected if and only if the observed or 
computed test statistic value falls in the rejection region (Devore, 2004). Evidence to 
reject a null hypothesis is given if the p-value is small enough (the probability that a 
difference of the magnitude observed may have occurred by chance). 

Since we were testing two different measures of effectiveness (MOE), two 
different null hypotheses were stated along with their corresponding alternative 
hypothesis. Our dependent variables, or MOEs, were the “probability of correct 
response” and its corresponding “time of response”. 

1, MOE “Probability of Correct Response” 
a. Null Hypothesis 

Hq : no difference between traditional and prototype display design 
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b. Alternative Hypothesis 

: prototype display design provides a higher percentage of correct 


response 

Null or alternative hypotheses can be explored by using the results of the binary 
data (1 for correct response and 0 for incorrect response) of the experiment when the test 
participant had to judge the current attitude. 

2, MOE “Time of Response” 

a. Null Hypothesis 

Hq'. no difference in response time between traditional and prototype 
display design 

b. Alternative Hypothesis 

H^ : prototype display enables to respond correctly in a shorter time 

The second null or alternative hypotheses can be explored by using the results of 
the time measurement in seconds and milliseconds of each judgement task. The time each 
participant required to judge his attitude, indicates how fast he can build his model of the 
current spatial situation and therefore being able to start a recovery if necessary. Each 
participant is tested on the traditional and the prototype display design. The order of 
presentation was counterbalanced. The data provides an indication about which display 
design enables a person to assess an aircraft attitude more rapidly. 

In addition, the post-experiment questionnaire provides us with feedback for both 
measures of effectiveness. Assuming a faster response to the judgement task when using 
the prototype display design, this should be reflected by selecting an answer of “border 
line”, “somewhat more” or “much more” with regards to better awareness of spatial 
orientation. 

Additionally, an answer to the question of the influence of a color coded support 
for pitch and/ or roll, either for the traditional or the prototype display design, can be 
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extracted from the questionnaire. By choosing any answer above “border line”, we can 
assume that the participant favoured an additional color support. 


3. Baseline Determination 

Before starting with any experiment, it is necessary to evaluate the number of 
participants necessary to achieve a statistically significant outcome. 

Kanji (1999) provides a general overview of the relation between a true or false 
hypothesis and when to reject or not to reject this hypothesis. The two ways of making 
mistakes when performing a hypothesis test are type I and type II errors. Type I errors 
represent mistakenly rejecting , when is true. On the other hand, an error of type 

II occurs when is false and it is not rejected. 

The goal is to avoid type I and type II errors and make the correct decision about 
accepting or rejecting the null hypothesis. 


What is true in the population? 


Conclusion No effect 

reached in a 

study 

Treatment effect 


Treatments have Treatments have 

no effect an effect 


Correct decision 

(P=l-y9) 

Type II error 

(p= Cl) 

Type I error 

(P = «) 

Correct decision 

(P=l-y9) 


Power 

Table 2. Outcomes of statistical tests (Murphy & Myors, 2004) 

The power of a statistical test is defined as the probability of rejecting the null 
hypothesis, when the alternative hypothesis is true (Montgomery & Runger, 2007). 
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Instead of simply calculating the level of power, it is also possible to determine 
the sample size to achieve specific levels of power. By convention, it is necessary to 
achieve at least a power of 0.05, otherwise the probability is greater than .05 that a Type I 
error will occur, i.e., failing to reject when is false. Achieving a power greater 
than 0.08 is always desirable (Murphy & Myors, 2004). 

We used a power analysis and determination of the sample size according to 
DeVeaux et ah, 2005. By regrouping the formula 


from: 


to: 


ME = t 


n-\ 


s 




ME 


n = 


/ , \2 
If • V ' 

yME j 


it is possible to determine the sample size. 


Explanation: ME: margin of error 

extent of an interval on either side 
: confidence interval 

an interval estimate for a population parameter 
s: standard error 

estimate of the standard deviation of a sampling 
distribution 

calculated by the product of p x q with p as the probability 
that the observed statistic value could occur and q as the 
probability that the observed statistic value could not occur 


To determine an appropriate sample size, we made the following assumptions. 
The level of significance will be a = 0.05 and the probability of 90% that the statistic 
value will occur. The results shown in Table 2 were calculated by using the above 
displayed formula in a spreadsheet in Microsoft Excel 2003. 
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ME 

P 

Q 

z 

Sample size n 

Cl 

0.03 

0.9 

0.1 

1.282 

14.79 

80% 

0.04 

0.9 

0.1 

1.645 

13.70 

90% 

0.04 

0.9 

0.1 

1.96 

19.45 

95% 

0.05 

0.9 

0.1 

2.326 

17.53 

98% 

0.05 

0.9 

0.1 

2.576 

21.50 

99% 


Table 3. Number of proposed partieipants 


Therefore we designed the experiment for 20 partieipants, assuming a margin of 
error of 0.04 with a p-value of 90% using a confidenee interval of 95%. Sinee we decided 
to choose a level of significance of a = 0.05, can be rejected if the p-value is less 
than 0.05. 


B. GENERAL DATA ANALYSIS 

1, General Data Description 

During the experiment 1,120 data points (20 participants x 4 trial x 14 tests = 
1,120) in different categories were collected. The following software was used for the 
data analysis: Microsoft Excel 2003 and SPSS 13.0 for Windows. These programs have 
also been used for the graphical output of the analysis. 

The data categories were: 

1) Initial Questionnaire: The questionnaire was also used to obtain 
demographic data like age in years and flight experience in hours. 

2) Judgement of the current attitude: The answer to the task of judging the 
current attitude as correct or incorrect. 

3) Judgement Time: The time the participant needed to judge the current 
attitude in sec. 

2) Post Experiment Questionnaire: The subjective assessment of the 

experiment. 
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2, Demographic Data 
a. Age 

Both pilots and non-pilots had a similar mean age. The standard deviation 
of the non-pilots was a little higher due to more participants of 40 years and older. The 
detailed statistics are displayed in Table 4. 



Mean 

STD 

Minimum 

Maximum 

Pilots 

35.50 

3.85 

28 

42 

Non-pilots 

34.00 

4.77 

28 

42 

Total 

34.75 

4.40 

28 

42 


Table 4. Statistics on participants’ age 


b. Flight Experience 


ID- 

Participant 

Pilot 

(1/0) 

Flight- 

Hours 

Mean 

Flight- 

Hours 

STD 

001 

1 

80.00 



002 

1 

20.00 

003 

1 

2000.00 

004 

1 

2500.00 

005 

1 

1400.00 

006 

1 

2400.00 

007 

1 

10.00 

008 

1 

1971.00 

101 

1 

2000.00 

102 

1 

1100.00 

1348.10 

943.29 


Table 5. Flight-Hours - Raw data 


Flight Hours 



ID - Participant 


Figure 31. Chart of individual flight-hours 
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Seven pilots out of the total of ten pilots had more than 1,000 flight hours 
and four out of these seven pilots had more than 2,000 flight hours, with one participant 
very close to the 2,000 flight hour barrier, one can consider the group of pilots as 
experienced. Most of them had some experience with SD during their flying career. Table 
6 provides the descriptive statistics of the pilot’s flight experience. 



Mean 

STD 

Minimum 

Maximum 

Flight experience 

1,348.10 

943.29 

10 

2,500 


Table 6. Statistics on pilot’s flight experience 


3, General Statistical Methodology 

We tested each participant on the factors “display design” and “color support/ no 
color support.” In addition, there were two types of participants, pilots and non-pilots, 
yielding a full factorial design with 2^ = 8 factor-level combinations. Our measures of 
effectiveness were correct answer of aircraft attitude (usual or unusual attitude) and the 
time a participant needed to make that judgment (see Figure 32 and Figure 33). 

Analysis of variance (ANOVA) was used to test the effects of the independent 
variables on the correct judgment of attitude and the time to respond. 
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Figure 32. Combination and evaluation of correct response 
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Figure 33. Combination and evaluation of time to judge 
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4. Baseline Comparison 

The basic data are displayed in the middle section of Figure 32 and Figure 33 and 
combined in Table 7 with the raw data. 


ID- 

Participant 

Pilot 

(1/0) 

orob. correct 

mean time 

(Traditional + 
Non-Color) 

orob. correct 

mean time 

(Traditional + 
Color) 

orob. correct 

mean time 

(Prototype + 
Non-Color) 

nrob. correct 

mean time 

(Prototype + 
Color) 

resDonse 

(Traditional + 
Non-Color) 

resDonse 

(Traditional + 
Color) 

resDonse 

(Prototype + 
Non-Color) 

resDonse 

(Prototype + 
Color) 

001 

1 

1.00 

1.44 

1.00 

0.66 

0.93 

0.91 

1.00 

0.68 

002 

1 

0.57 

3.07 

1.00 

1.28 

1.00 

2.22 

1.00 

1.31 

003 

1 

0.86 

1.74 

0.79 

2.32 

1.00 

1.54 

1.00 

1.45 

004 

1 

1.00 

1.68 

1.00 

1.19 

1.00 

1.06 

1.00 

1.80 

005 

1 

0.86 

2.02 

1.00 

0.77 

1.00 

1.30 

1.00 

0.72 

006 

1 

0.93 

1.91 

1.00 

\n 

1.00 

1.05 

1.00 

1.15 

007 

1 

0.86 

1.44 

1.00 

1.63 

1.00 

1.07 

1.00 

1.03 

008 

1 

1.00 

1.70 

1.00 

1.74 

1.00 

1.93 

1.00 

1.20 

101 

1 

1.00 

1.62 

1.00 

2.28 

1.00 

1.40 

1.00 

0.89 

102 

1 

1.00 

2.29 

1.00 

2.02 

1.00 

2.00 

1.00 

1.36 

201 

0 

1.00 

2.31 

1.00 

1.74 

1.00 

1.83 

1.00 

1.04 

202 

0 

0.93 

1.46 

0.86 

1.88 

0.93 

1.25 

1.00 

0.83 

203 

0 

0.86 

2.02 

1.00 

0.88 

1.00 

1.57 

1.00 

0.96 

204 

0 

0.86 

2.22 

0.93 

1.22 

0.86 

1.79 

1.00 

0.84 

205 

0 

1.00 

3.06 

1.00 

3.27 

1.00 

2.20 

1.00 

1.84 

206 

0 

0.71 

4.89 

1.00 

0.98 

1.00 

1.74 

1.00 

1.19 

207 

0 

0.93 

2.40 

1.00 

1.10 

1.00 

1.83 

1.00 

0.79 

208 

0 

0.86 

3.02 

1.00 

0.96 

1.00 

1.38 

1.00 

0.89 

209 

0 

1.00 

2.83 

1.00 

1.03 

1.00 

2.21 

1.00 

0.90 

210 

0 

0.86 

3.85 

1.00 

1.03 

0.93 

2.19 

1.00 

0.91 


1 MEAN : II 

0.90 

JL 

2.35 

Jl 

0.98 

JL 

1.49 

II 

0.98 

II 

1.62 

II 

1.00 

II 

1.09 1 


1 STD : ir 

0.11 

□□ 

0.87 

□1 

0.05 

□L 

0.63 

□L 

0.04 

□L 

0.42 

□L 

0.00 

□L 

0.32 1 


Table 7. Baseline comparison - Raw data 


C. ANALYSIS OF VARIANCE 
1, General Linear Model 

Analysis of variance (ANOVA) is an analysis method for testing equality of 
means across three or more treatment groups. In our case we used a one-way ANOVA for 
repeated measures. On one hand we have a “within subjects” (see Table 8 and Table 9) 
design because every participant received the same treatment. 
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Within-Subjects Factors 


Measure: MEASURE 1 


display color 

Dependent 

Variable 

1 1 

2 

prob. 

correctrespo 

nse 

Traditional 

NonColor 

prob. 

correctrespo 

nse 

Traditional 

Color 

2 1 

2 

prob. 

correctrespo 

nse 

Prototype 

NonColor 

prob. 

correctrespo 

nse 

Prototype 

Color 


Table 8. Within subjects for probability of correct response 


Within-Subjects Factors 


Measure: MEASURE 1 


display color 

Dependent 

Variable 

1 1 

2 

meantime 

Traditional 

NonColor 

meantime 

Traditional 

Color 

2 1 

2 

meantime 

Prototype 

NonColor 

meantime 

Prototype 

Color 


Table 9. Within subjects for mean time 
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The two groups of participants, pilots and non-pilots, were a “between subjects” 


factor (see Table 10). 


Between-Subjects Factors 



N 

Pilot 

0 

10 

(1/0) 

1 

10 


Table 10. Between subjects 


Therefore we are able to assess the effect of display design or color versus the 
status as pilot or non-pilot. 


2, ANOVA for Probability of Correct Response 

The ANOVA for probability of correct response gave us the statistical output 
shown in Table 11. 


Tests of Within-Subjects Contrasts 


Measure: MEASURE_1 


Source 

display 


Type III Sum 
of Squares 

df 

Mean Square 

F 

Siq. 

display 

^SBM 


.049 

1 

.049 

11.130 

.004 

display * PilotIO 

^SBM 


.000 

1 

.000 

.056 

.816 

Error(display) 

^SBM 


.079 

18 

.004 



color 


Linear 

.042 

1 

.042 

10.805 

.004 

color * PilotIO 



.001 

1 

.001 

.250 

.623 

Error(color) 



.071 

18 

.004 



display * color 

Linear 


.016 

1 

.016 

3.540 

.076 

display * color * PilotIO 

Linear 


.000 

1 

.000 

.053 

.820 

Error(display*color) 

Linear 


.083 

18 

.005 




Table 11. Test of within-subjects contrasts for probability of correct response 


The effect of display design (traditional or Weber Box) was highly significant, F 
= 11.13, p = 0.004. Likewise, the effect of color coding was significant, F = 10.81, p = 
.004. There were no significant two-way interactions between display design and color (p 
= 0.076), display design and pilot/ no pilot (p = 0.816), or color and pilot/ no pilot (p = 
0.623). Similarly, the three way interaction between display, color, and pilot was not 
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significant (p = 0.820). Figure 34 shows the nearly-significant interaction between 
display type and color. It can also be seen in the figure that response accuracy was higher 
for the WEBER Box than for the traditional display, both with and without color. 


Estimated Marginal Means of MEASURE_1 



color 

- no color 

- color 


Eigure 34. Interactions of display design and color 


Participants using the WEBER-Box reached a probability of correct response of 
98% without color coding and 100% with color coding. These results indicate that high 
accuracy in judging attitude is achieved using the WEBER Box. 


a. T-Test: Traditional Design Versus Color 

In support of the ANOVA findings, a t-test was performed to determine 
whether color coding had an effect on response accuracy with the traditional display (see 
Table 12). The significant result, p < .05, indicates that response accuracy using the 
traditional display was better with color coding than without it. The boxplot also supports 
this conclusion (see Eigure 35). 
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prob. correct response 
(Traditional without color) 

prob. correct response 
(Traditional with color) 

Mean 

0.9045 

0.979 

Variance 

0.012552368 

0.003146316 

Observations 

20 

20 

P(T<=t) one-tail 

0.005702026 


P(T<=t) two-tail 

0.011404052 



Table 12. Analysis in the probability of correct response between traditional 

design without color vs. with color 



prob. correct response (Traditional + Non-Color) prob. correct response (Traditional + Color) 


Figure 35. Graphical output of the analysis in the probability of 
correct response between traditional design 
without color vs. with color 


3, ANOVA for Response Time 

The ANOVA for response time resulted in the statistical output shown in Table 
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Tests of Within-Subjects Contrasts 


Measure: MEASURE_1 


Source 

display 

color 

Type III Sum 
of Squares 

df 

Mean Square 

F 

Sip. 

display 

Linear 


6.311 

1 

6.311 

58.699 

.000 

display * PilotIO 

Linear 


.374 

1 

.374 

3.479 

.079 

Error(display) 

Linear 


1.935 

18 

.108 



color 


Linear 

9.737 

1 

9.737 

23.978 

.000 

color * PilotIO 


Linear 

3.054 

1 

3.054 

7.520 

.013 

Error(color) 


Linear 

7.310 

18 

.406 



display * color 

Linear 

Linear 

.533 

1 

.533 

1.951 

.179 

display * color * PilotIO 

Linear 

Linear 

.422 

1 

.422 

1.544 

.230 

Error(display*color) 

Linear 

Linear 

4.918 

18 

.273 




Table 13. Test of within-subjects contrasts for mean response time 


Display type (traditional versus WEBER Box) had a highly significant effect on 
mean response time, E = 58.70, p < 001. Similarly, color coding had a highly significant 
effect on response time, E = 23.98, p < .001. 

The two way interaction of display type and pilot/ non-pilot did not reach 
significance, E = 3.48, p = .079, nor did the interaction of display type and color coding, 
E=1.95,p = .179. On the other hand, the interaction of color coding and pilot/ non-pilot 
was significant, E = 7.52, p < .05. The three way interaction of display type, color coding, 
and pilot was not significant, E = 1.54, p = .230. 

A graphical representation of display type and color coding is given in Eigure 36. 
It can be seen in the figure that response times were shorter with the new display design 
(WEBER Box) than with the traditional instrument design, and that held true for both 
color coded and no color coded displays. 


58 




Estimated Marginal Means of MEASURE_1 



color 

- no color 

- color 


Figure 36. Non-significant interaction between display design and color 


The mean time decreased when switching from the traditional display to the 
WEBER-Box. Adding color to the displays made this decrease even more obvious. 

Eigure 37 shows the significant interaction between display design and pilot/ non¬ 
pilot. Both pilots and non-pilots responded more quickly with the WEBER Box than with 
the traditional instrument design, but the advantage of the WEBER Box was greater for 
the non-pilots. This result is not surprising because pilots have considerable experience 
with the traditional display design, whereas the non-pilots were quite slow in responding 
to the traditional display. 


59 



Estimated Marginal Means of MEASURE_1 



Pilot (1/0) 

- no pilot 

-pilot 


Figure 37. Interactions between color and pilots 


This graphical outcome shows at once that the non-pilots profited most from the 
WEBER-Box. Both, pilots and non-pilots, could decrease their mean response time when 
tested on the WEBER-Box. But the non-pilots mean response time dropped in such a 
significant way even underneath the pilots value when using the WEBER-Box. 

This is an additional proof of the superiority of the WEBER-Box. On the other 
hand there is room fur further investigations, why the pilots didn’t benefit as much from 
the WEBER-Box as the non-pilots. 


a. T-Test: Pilot Versus Display Design 

In support of the ANOVA findings a /-test was performed to determine 
whether display design had an effect for mean response time on pilots (see Table 14). The 
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significant result, p < .05, indicates that the mean response time of the pilots using the 
WEBER-Box was better than with the traditional design. The boxplot also supports this 
conclusion (see Eigure 38). 



mean time 

(Pilot with trad, design) 

mean time 

(Pilot with WEBER-Box) 

Mean 

1.729 

1.329 

Variance 

0.136054444 

0.123676667 

Observations 

10 

10 

P(T<=t) one-tail 

0.011579295 


P(T<=t) two-tail 

0.023158589 



Table 14. Analysis of the response time between pilots with the 
traditional design vs. the WEBER-Box 



Eigure 38. Graphical output of the analysis of the mean response time 

for pilots with the traditional design 
vs. the WEBER-Box 


In support of the ANOVA findings another t-test was performed to determine 

whether display design had an effect for mean response time for non-pilots (see Table 
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15). The significant result, p < .05, indicates that the mean response time of the non-pilots 
using the WEBER-Box was much better than with the traditional design. The boxplot 
also supports this finding (see Eigure 39). 



mean time 
(Non-Pilot with trad, 
design) 

mean time 

(Non-Pilot with WEBER- 
Box) 

Mean 

2.109 

1.407 

Variance 

0.32061 

0.074445556 

Observations 

10 

10 

P(T<=t) one-tail 

0.001190929 


P(T<=t) two-tail 

0.002381859 



Table 15. Analysis of the response time for non-pilots with the 
traditional design vs. the WEBER-Box 



Eigure 39. Graphical output of the mean response time (sec) 
for non-pilots with the traditional design 
vs. the WEBER-Box 
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b. T-Test: Pilot Versus Color 

In support of the ANOVA findings a t-test was performed to determine 
whether color had an effect on response time for pilots (see Table 16). The significant 
result, p < .05, indicates that the mean response time of the pilots using the WEBER-Box 
was faster than with the traditional design. The boxplot also supports this finding (see 
Eigure 40). 



mean time 

(Non-Pilot without color) 

mean time 
(Pilot with color) 

Mean 

2.305 

1.214 

Variance 

0.336538889 

0.249982222 

Observations 

10 

10 

P(T<=t) one-tail 

0.000137018 


P(T<=t) two-tail 

0.000274036 



Table 16. Analysis of the response time for non-pilots with 
color vs. without color 



Eigure 40. Graphical output of the response time (sec) of non-pilots 

with color vs. without color 
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D, POST-EXPERIMENT QUESTIONNAIRE 

After the primary data collection was finished, each participant was asked to fill 
out the post-experiment questionnaire. This questionnaire served as an indicator of the 
attitude of the participants toward the two different display designs and the difference of 
a colored-coded support. It also provided subjective data in relation to the correct/ 
incorrect answers and the time each participant needed to judge the various aircraft 
attitudes. The raw data from the questionnaire can be seen in Table 17. 



Difficulty of 
Task 

Difficulty 
judging by 
Traditional 

Difficulty 
judging by 
Prototype 

Beneflts of 
Color-coding 

Better aware 

with 

Prototype 
without Color 

Better aware 

with 

Prototype 
with Color- 
Coding 

Prototype in- 
/ decrease 
monitoring 
demands 

Coding 


1-5 

1-5 

1-5 

1-5 

1-5 

1-5 

1-5 

001 

3.00 

3.00 

2.00 

1.00 

4.00 

3.00 

2.00 

002 

3.00 

4.00 

2.00 

1.00 

4.00 

4.00 

2.00 

003 

2.00 

2.00 

2.00 

1.00 

3.00 

4.00 

2.00 

004 

1.00 

2.00 

1.00 

2.00 

4.00 

4.00 

2.00 

005 

4.00 

4.00 

2.00 

2.00 

5.00 

4.00 

1.00 

006 

1.00 

1.00 

1.00 

1.00 

3.00 

3.00 

3.00 

007 

2.00 

3.00 

1.00 

1.00 

4.00 

5.00 

2.00 

008 

1.00 

2.00 

2.00 

1.00 

1.00 

1.00 

2.00 

101 

2.00 

3.00 

2.00 

1.00 

4.00 

4.00 

2.00 

102 

1.00 

2.00 

2.00 

1.00 

3.00 

3.00 

4.00 

201 

2.00 

3.00 

1.00 

1.00 

5.00 

4.00 

1.00 

202 

4.00 

5.00 

4.00 

1.00 

4.00 

4.00 

3.00 

203 

3.00 

3.00 

3.00 

1.00 

5.00 

3.00 

3.00 

204 

2.00 

4.00 

1.00 

1.00 

5.00 

5.00 

1.00 

205 

3.00 

4.00 

2.00 

1.00 

4.00 

4.00 

3.00 

206 

2.00 

4.00 

1.00 

1.00 

5.00 

5.00 

5.00 

207 

3.00 

4.00 

2.00 

1.00 

4.00 

5.00 

1.00 

208 

2.00 

4.00 

1.00 

1.00 

5.00 

4.00 

4.00 

209 

2.00 

3.00 

3.00 

1.00 

5.00 

3.00 

2.00 

210 

3.00 

3.00 

5.00 

1.00 

2.00 

5.00 

5.00 


Table 17. Post-Experiment Questionnaire - Raw data 

We calculated three different types of percentages, as shown below, to help reveal 
the participants’ opinions: 
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overall percentage 


30% 

20%/ 40% 

/ \ 

percentage of pilots percentage of non-pilots 

Figure 41. Post questionnaire calculation 


Then we transferred these percentages again into the questionnaire (see Table 18). 
By doing it this way, we received a first synopsis of the summarized subjective 


impression of all participants. 


Question 


Answer-Op 

tions 


In general, how do you judge the 

Very 

Somewhat 

Border 

Somewhat 

Very difficult 

difficulty of evaluating the current 
attitude of the aircraft? 

easy 

easy 

line 

complicated 



20% 

40% 

30% 

10% 

0% 


(40%/0%) 

(30%/50%) 

(20%/40%) 

(5%/5%) 

(0%/0%) 

In general, how do you judge difficulty 

Very 

Somewhat 

Border 

Somewhat 

Very difficult 

of using the traditional flight instruments 
for this task? 

easy 

easy 

line 

complicated 

5% 


5% 

20% 

35% 

35% 


(10%/0%) 

(40%/0%) 

(30%/40%) 

(20%/50%) 

(0%/10%) 

In general, how do you judge the 

Very 

Somewhat 

Border 

Somewhat 

Very difficult 

difficulty of using the WEBER-Box for 
this task? 

easy 

easy 

line 

complicated 



35% 

45% 

10% 

5% 

5% 


(30%/40%) 

(70%/20%) 

(0%/20%) 

(0%/10%) 

(0%/10%) 

In general, how do you judge the 

Very 

Somewhat 

Border 

Somewhat 

Very poor 

benefits of color coding of an unusual 
attitude? 

good 

good 

line 

poor 



90% 

10% 

0% 

0% 

0% 


(80%/100%) 

(20%/0%) 

(0%/0%) 

(0%/0%) 

(0%/0%) 

Did you feel better aware of the spatial 
orientation of the airplane with the 
WEBER-Box in comparison with the 

Much less 

Somewhat 

less 

Border 

line 

Somewhat more 

Much more 

traditional flight instruments, when both 

5% 

5% 

15% 

40% 

35% 

are without color coding of an unusual 
attitude? 

(10%/0%) 

(0%/10%) 

(30%/0%) 

(50%/30%) 

(10%/60%) 

Did you feel better aware of the spatial 
orientation of the airplane with the 
WEBER-Box in comparison with the 

Much less 

Somewhat 

less 

Border 

line 

Somewhat more 

Much more 

traditional flight instruments, when both 

5% 

0% 

25% 

45% 

25% 

are with color coding of an unusual 
attitude? 

(10%/0%) 

(0%/0%) 

(30%/20%) 

(50%/40%) 

(10%/40%) 

Did having the WEBER-Box increase or 

Greatly 

Somewhat 

Unaffected 

Somewhat 

Greatly 

decrease your monitoring demands in 
comparison with the traditional flight 

decreased 

decreased 

20% 

increased 

increased 

instruments? 

20% 

40% 

10% 

10% 


(10%/30%) 

(70%/10%) 

(10%/30%) 

(10%/10%) 

(0%/20%) 


Table 18. Results of the post-experiment questionnaire 
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This subjective data provides initial evidence about the preference and acceptance 
of the two display designs and the opinion of the participants towards the option of a 
color-coded feature for detecting unusual aircraft attitudes. 

The difficulty using the traditional display design forjudging an attitude was rated 
as “border line” or “somewhat complicated” by 70% of the participants. The majority of 
this 70% are non-pilots, which is not surprising since pilots are trained with this 
instrument. Therefore, the pilots considered the use of the traditional display design 
relatively easy. On the other hand, 80% of all participants rated the use of the prototype 
display design as “easy” or “somewhat easy.” Interestingly, all of the pilots came to the 
same conclusion. This gives strong evidence that pilots as well as non-pilots 
acknowledge ease of use of the prototype design and felt that it supports their task of 
judging different attitudes. 

The questionnaire data are quite clear regarding the use of color-coding for the 
judgment of unusual attitudes. All of the participants acknowledged the advantage of 
color-coding. 

Over 70% of the participants reported that they were more aware of their spatial 
orientation with the prototype design (WEBER Box), independent of whether it was 
color-coded or not. This result was found for pilots as well as for non-pilots. 
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VII. CONCLUSIONS 


A. OVERALL ASSESSMENT OF THE STATISTICAL RESULTS 

The main question for an overall assessment was whether a significant difference 
between the display combinations could be determined. 

In general, the effect of display design (traditional or WEBER-Box) and the effect 
of color coding was significant. Specifically for the probability of correct response, 
participants using the WEBER-Box reached a probability of correct response of 98% 
without color coding and 100% with color coding. The mean time also decreased when 
switching from the traditional display to the WEBER-Box. Adding color to the displays 
made this decrease even more obvious. Response times were always shorter with the 
WEBER-Box than with the traditional display design, and that held true for both color 
coded and non color coded display. Therefore the WEBER-Box was better than the 
traditional display for both accuracy of determining unusual attitudes and speed (response 
time). 

The outcomes also must be considered in light of the “pilot, non-pilot” variable. 
Since both pilots and non-pilots participated, one has to take into consideration that pilots 
would be expected to perform better. They are not only familiar with the traditional 
display design, but they are also trained specifically on this kind of task, judging an 
attitude and then starting a recovery operation when necessary. This also enhances their 
skills to read and interpret flight instruments. Due to time constraints on the other hand, 
there was very little time to familiarize the non-pilots with the different display designs 
and train them for the task of judging an aircraft attitude. 

The answers in the post-experiment questionnaire confirmed the outcome of the 
statistical analysis. As mentioned in the previous chapter, there is agreement between the 
subjective response of the participants in the post-experiment questionnaire and the 
objective data. Eor example, both pilots and non-pilots felt that adding color to each of 
the displays increased their ability for spatial awareness and spatial orientation. 80% of 
the pilots and 100% of the non-pilots judged the color-coding to be beneficial. 
Comparing the traditional display design with the prototype display design, more than 
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70% of the participants preferred the new design, independent of adding color or not, 
compared to the old design. This result gains importance because 60% of the pilots 
judged an improved awareness with the prototype design as “somewhat more” or “much 
more”. 

A detailed look into the data shows that not only the probability of correct 
answers increased, but even more important, the time required to judge the current 
attitude decreased significantly. For example the mean response times changed as 
follows: 

• traditional to prototype: from 1.92 sec to 1.37 sec (29% reduction) 

• without color to color: from 1.99 sec to 1.30 sec (36% reduction) 

• traditional without color to traditional with color: from 2.35 sec to 1.49 sec 
(37% reduction) 

• prototype without color to prototype with color: from 1.62 sec to 1.09 sec 
(33% reduction) 

By the results of the ANOVA and the additional Mests, one can see that the 
prototype design enabled the participants to judge the situation faster and more 
accurately. These results clearly show the superiority of the prototype display design 
(WEBER Box). 

B, DISCUSSION AND CONCLUSION 

To understand the findings of this study, one must consider the purpose of the 
new prototype design. It was built to support pilots or operators of UAVs in extreme 
situations. With the findings of the research questionnaire, we were able to identify a 
current issue of the UAV community, in our case of the Predator UAV. That is, the 
WEBER-Box would help in the recognition of the current aircraft attitude, facilitate 
recovery when an unusual attitude existed, and help to prevent misperception of an 
unusual attitude. Based on this information, we developed the experiment and compared 
the traditional display design with the new design (WEBER Box). We also evaluated the 
color factor and its two levels, with or without color-coding. 
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In the experimental task, the participants judged their spatial attitude in a static 
environment. Based on the level of significance of « = 0.05, the outcome was 
remarkably obvious. Using the WEBER Box, performance increased in both accuracy 
and response time. Eor the dependent variable “correct response,” using the prototype 
display design improved performance by 105%. When using the color-coded feature, the 
same 105% gain of performance was found. The results are even more significant for the 
dependent variable “mean time to respond” where the performance improved by 145% 
using the new prototype design. 

The value of the prototype display also was supported by the post-experiment 
questionnaire and by comments and suggestions as the final feedback after completing 
the experiment. Most of the participants stated that the new display design was very 
helpful in judging the aircraft attitude task. They felt an improvement in their spatial 
awareness and the new design helped them to decrease the time for correctly judging 
their attitude. Thus, the prototype display design made it easier to accomplish their task. 
By adding color to the displays, it was even easier to judge the current attitude and the 
percentage of correct responses increased to almost 100% or even reached a level of 
100% in the case of the new design. Eikewise, the time to achieve a correct response 
decreased significantly too. 

Therefore the combination of the new prototype design together with color-coding 
showed a clear increase of performance compared to the traditional display design 
without color that is currently being used. 


C. FUTURE RESEARCH 

As is often the case in research, in addition to answering some questions, other 
questions and suggestions have been raised. Most of them fall in the category of 
suggested design improvements, in our case design improvements of the new prototype 
design in combination with the color support. These can be summarized as follows: 

• What is the ideal size of the new prototype design? Since it provides a 
numerical output of pitch and roll, the question is how small could it be to 
still fit in the environment of a ground control station of UAV but with a 
font size still large enough to read the numbers. 
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• Is it possible to decrease the time for judging the attitude by placing the 
digital readout of pitch and roll to a more central location in the display? 
Currently there is still an eye movement necessary from the animated 
aircraft to the digital pitch and roll values, since it is not possible to see all 
of the information with one view. 

• Would it be better to change the color of the entire aircraft avatar instead 
of the output of pitch and roll digital display, when reaching an unusual 
attitude? 

• Can the output pitch and roll be represented in a different way so that the 
aircraft avatar and the output can be seen at the same time? A possible 
solution to this question is, giving the new prototype display design a 
background like a grid. The aircraft avatar would “fly” on top of the grid 
and pitch and roll as additional numerical output could be displayed 
sideways to this grid. 
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APPENDIX A. RESEARCH QUESTIONNAIRE 


Naval Postgraduate School, LTC Thomas Zirkelbach, German Army Jan 2007 

Background Information 

Spatial awareness and the skills of an operator to transition from the current spatial 
attitude into the necessary spatial orientation and position mainly depend on human 
performance in spatial orientation tasks. Degraded spatial awareness, particularly in 
extreme flight situations, may lead to loss of equipment and to lower operational 
effectiveness. How to maintain and improve spatial awareness is one of the major issues 
in complex 3D environments. In particular, orientation tasks in 3D environments with 
many degrees of freedom are very difficult to accomplish. 

Based on these findings Maj Axel Weber (GE) has developed in his thesis “Application 
of Avatars in Display Design to support Spatial Awareness in Extreme Elight Situations” 
a prototype display design that helps to recognize and recover from an unusual attitude 
faster and more accurate compared to a traditional display design. 



Screenshot of the new prototype display design, called WEBER-Box 


I am expanding this work researching with my thesis “Pictorial Display Design to 
enhance Spatial Awareness of Operators in Unmanned Aviation” possible applications of 
the WEBER-Box for UAVs. 

Therefore I am currently looking for UAV incidents which will be the initial position for 
an experiment that should have a realistic background. Therefore I am assuming that 
operations of large UAVs are normally not flown in unusual attitude, but nevertheless 
occasionally it occurs. One basic question is “How often do the operators get unintended 
into an unusual attitude?” Thus I have elicited and focused through my research on two 
general approaches. 

Approach 1: Recovery from an unusual attitude 

Currently available literature states that incidents occur if the data link is interrupted, will 
be reestablished and a recovery of the UAV is still not possible. I am also thinking of a 
different situation where there are no problems with the data-link or the GPS-Signal but 
the internal computer which is responsible for the stabilization system fails and the UAV 
e. g. Predator has to be recovered manually by the operator in the Ground Control 
Station. In such a case I assume the UAV will be in an unusual attitude before the 
recovery even starts. 
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Approach 2: Launch and Recovery 

With this approach I focus on the topic how the WEBER-Box could support recovery 
operations. 


Naval Postgraduate School, ETC Thomas Zirkelbach, German Army Jan 2007 

Questionnaire 


1) We have a prototype display design that helps to recognize and recover from an 
unusual attitude. 

a) Can you think of a situation where it can be used? 

2) a) Have you experienced situations of unusual attitude? 

b) Which UAV has been involved? 

c) What was the cause for this unusual attitude? 

d) How did you recover it, autonomously or manually? 

e) If manually, which instruments did you use? 

f) Have you been successful? 

3) a) Have you experienced situations of unusual attitude when handing over a UAV 

from one Ground Station to another? 

b) If yes, which UAV has been involved? 

c) Please describe: 


4) a) Have you experienced situations of unusual attitude after an interrupted data link 
was restored? 

b) If yes, which UAV has been involved? 

c) Please describe: 


5) Assume the following scenario: a Predator has no data-link interruption and the GPS 
signal works, but its internal computer responsible for its autonomous flying fails. 

a) Have you ever experienced a situation like this? 

b) Does the operator take over in remote control? 

c) Did this situation cause an unusual attitude? 


6) Can you launch and recover in instrument conditions? 
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7) If you had to accomplish evasive maneuvers to escape anti-aircraft fire, 

a) Would you do this in remote control? 

b) Which flight instruments would you be using? 

c) How difficult would this be? 
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APPENDIX B. IRB APPLICATION 



Michael E. McCauley, Ph.D. 
Operations Research Department 
Glasgow Hall 
Naval Postgraduate School 
Monterey, California 93943 


831-656-2191 
DSN: 756-2191 
Fax: 831-656-2595 
memccaul@nps.edu 


To: Protection of Human Subjects Committee 


Subject: Application for Human Subjects Review for 

Pictorial Display Design to enhance Spatial Awareness of Operators in 
Unmanned Aviation 


1. Attached is a set of documents outlining a proposed experiment to be conducted over 
the three month to support the thesis of LTC Thomas Zirkelbach, GE. 

2. We are requesting approval of the described experimental protocol. An experimental 
outline is included for your reference that describes the methods and measures we 
plan to use. 

3. We include the consent forms, privacy act statements, all materials and forms that a 
subject will read or fill-out, and the debriefing forms (if applicable) we will be using 
in the experiment. 

4. We understand that any modifications to the protocol or instruments/measures will 
require submission of updated IRB paperwork and possible re-review. Similarly, we 
understand that any untoward event or injury that involves a research participant will 
be reported immediately to the IRB Chair and NFS Dean of Research. 


Michael E, McCauley 
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APPLICATION FOR 
HUMAN SUBJECTS REVIEW 


(HSR) 


NPS IRB NUMBER 
RSPO) _ 


(to be assigned by 


PRINCIPAE INVESTIGATOR(S) (Full Name, Code, Telephone) 

Michael E, McCauley, Ph,D,, Monterey, CA 93943, Phone: 831 656-2191 
(Thesis of ETC Thomas Zirkelhach, GE) 


TITEE OE EXPERIMENT 

Does an outside-in aircraft animation display help UAV operators to accurately perceive 
unusual attitudes? 


APPROVAE REQUESTED [X] New [ ] Renewal [ ] Amendment 


EEVEE OE RISK [ ] Exempt [X] Minimal [ ] More than Minimal 
Justification: Since there is no wide field of view (FOV) terrain display, simulator sickness 
is extremely unlikely. A prior study using this prototype display found zero incidence of 
simulator sickness. 


WORK WIEE BE DONE IN (Site/Bldg/Rm) 

NPS, Watkins Hall, RM 212B 

ESTIMATED NUMBER OF DAYS TO 
COMPEETE 

30 

MAXIMUM NUMBER OF SUBJECTS 

50 

ESTIMATED EENGTH OF EACH 
SUBJECT’S PARTICIPATION 

2 hours (1 hour in flight simulation) 


SPECIAE POPUEATIONS THAT WIEE BE USED AS SUBJECTS 


[ ] Subordinates [ ] Minors [X] NPS Students [ ] Special Needs (e.g. Pregnant women) 

Specify safeguards to avoid undue influence and protect subject’s rights: 

Participant Consent Form & Minimal Risk Statement 


SCIENTIFIC MERIT REVIEW (Check all that apply) 

[ ] This research is part of a funded project (Job Order Number ) 

[X] This research is a student thesis (Attach a copy of the approved thesis proposal) 

[ ] Other (Attach a complete research proposal) 


OUTSIDE COOPERATING INVESTIGATORS AND AGENCIES 
None 

[ ] A copy of the cooperating institution’s HSR decision is attached. 
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DESCRIPTION OF RESEARCH (attach additional sheet if needed). 

Methodology attached 

I have read and understand NPS Notice on the Protection of Human Subjects. If there are any 
changes in any of the above information or any changes to the attached Protocol, Consent 
Form, or Debriefing Statement, I will suspend the experiment until I obtain new IRB approval. 

SIGNATURE_ DATE_ 
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Methodology 


Experiment Goals and Purpose 

This research will investigate the effectiveness of a prototype display for pilot/operator 
recognition of aircraft/UAV attitude. 

Research Questions 

Will the new display design help operators to recognize aircraft attitude accurately? 

Will the new design help in discriminating between a normal and unusual attitude? 

Will the new design help prevent misperception of an unusual attitude? 

Will the new design reduce errors in recovery from unusual attitudes? 

Will the new design support the substitution of operators for trained pilots in unmanned 
aviation tasks? 

What will be the level of user acceptance of operators and pilots regarding the new 
design? 

Participants 

The participants will be randomly assigned on a volunteer basis from students and faculty 
of NPS. The study will involve at the most 50 participants. It is intended to split them 
equally into 25 pilots and 25 non-pilots. 

Apparatus 

The experiment will take place at the Naval Postgraduate School, Monterey, California, 
Watkins Building, room 212B. Two workstations will be used. Workstation 1 is for 
instruction and training while Workstation 2 will be established for the underlying 
experiment. 

Workstation 1; 

The computer-generated images will be displayed on a 19” computer monitor with a 1024 
X 768 pixel resolution. The pictures of the flight situations were generated by screenshots 
of the commercial flight simulator software X-Plane'^'^, version 7.13 and by screenshots of 
the WEBER-Box program. The pictures were stored in a PowerPoinE'^ slideshow to ensure 
that every participant will see the different aircraft attitudes in the same order. This 
slideshow will be used to explain the differences between a usual and unusual attitude. The 
simulation of the flight situations also will be generated by X-Plane'^'^ and by the WEBER- 
Box program. The predefined simulations of usual and unusual attitude will be stored in X- 
Plane'^'^ to help the participants to gain a better understanding of this complex situation in a 
dynamic environment. 

Workstation 2; 

The computer-generated images will be displayed on two 19” computer monitors with a 
1024 X 768 pixel resolution. The two screens will be located on top of each other, similar to 
the ground station of a Predator UAV. The simulation of the flight situations will be 
generated by the commercial flight simulator software X-Plane'^'^, version 7.13 and by the 
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WEBER-Box program. The predefined simulations of usual and unusual attitude will be 
stored in X-Plane'^’^ to ensure that every participant will see the different attitudes in the 
same order. 

Procedures 

The overall experiment will be subdivided into four phases. 

Phase pne - Preparatipn: 

The participant will be asked to fill out the necessary paperwork, including a consent form, 
a privacy act statement, a minimal-risk information form, and an initial/ demographic 
questionnaire. 

Phase two - Introduction and Training 

The participant will be introduced to the experiment, receiving information about the UAV, 
the task as an operator of the UAV, and details about usual and unusual attitudes. The 
participant also will watch a 10-minute introduction video of usual and unusual attitudes as 
displayed by the traditional flight instruments and by the WEBER-Box. This pre- 
experimental period, phase one and two, will take approximately 25 minutes. 

Phase .three - Experiment 

The participant will start with the first experimental sub-task. Each of his or her 14 trials 
will be scheduled for 120 seconds. The first two trials will be practice trials. The 
experimenter will start the prepared simulation of several flight situations that end in a 
usual or unusual attitude and, after 120 seconds, the simulation will stop with a picture of 
flight instruments in a certain flight situation. The participant will judge the “frozen” 
situation as a usual or unusual attitude. Then, the experimenter will initiate the next trial 
until the participant has finished twelve trials, plus two test trials. 

After accomplishing the first block of experiments, the participant will take a 5-minute 
break. Then the participant will start the second experimental task. All the participants will 
be exposed to all of the instrument setups and all of the sub-experiments (a within-subjects 
design). Each participant will be presented with seven trails of usual attitude with 
traditional design and seven trails of unusual attitude with the traditional design. Then the 
same procedure will apply with the new display design, the WEBER Box. 

Phase four - .Questionnaire 

When finished, the participant will be asked to fill out the post-experimental questionnaire. 
This completes the experimental session. 
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APPENDIX C. IRB APPROVAL 



Naval Postgraduate School 

Institutional Review Board (IRB) 


22-Feb-07 


From: 

To: 


LT Brent Olde, Ph.D. 

Research Profeasor Michael McCauley 


Subject: YOUR PROJECT: PICTORIAL DISPLAY DESIGN TO ENHANCE 

SPATIAL AWARENESS OF OPERATORS IN UNMANNED 
AVIATION 

1. The NPS IRB i( pleased to inform you that the NPS Institutional Review 
Board has approved your project (NPS IRB# NPS2007CK)42). 

2. The NPS IRB was originally certified by BUMED on 26 July 2002 and 
has been re-certified until 30 March 2007. 

3. This approval is valid for one year from this date. Please submit a copy of 
all records and consent forms to the Research aixl Sponsored Programs 
Office (Laura Ann Small, Halligan Hall, Room 201B) at the conclusion of 
this project. 

4. If your protocol changes at any time, you will need to resubmit your 
project proposal to the NPS IRB. 


Sincerely, 



Lt Brent Olde, Ph D. 

Chair 

NPS Institutional Review Board 
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APPENDIX D. INTRODUCTION / TASK DESCRIPTION 



Pictorial Display to Enhance Spatial 


Awareness of Operators in 
Unmanned Aviation 

bii 


LTC Thomas Zirkelbach 
German Army 



, m 88 Thesis Experiment 

V 


Thank you participating in this 
experiment. Without your support and 
your dedication I would never be able 
to accomplish my mission finishing 
this experiment and therefore my 
thesis. 

Thank you very much! 


Note: To continue, please push the return key. 
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Thesis Experiment 


V 


The experiment is divided into the 
following phases: 


Phase 1: 

Pre-Paperwork 


Phase 2: 

Introduction 



Training 


Phase 3: 

Experiment 


Phase 4: 

Post-Paperwork 

3 


Thesis Experiment 


This PowerPoint presentation will guide 
you through Phase 1 (Pre-Paperwork), 
Phase 2 (Introduction and Training), and 
the beginning of Phase 3 (Description of 
the Experiment). 

As soon as you are done we will start the 
experiment. 

Do not hesitate to ask me questions 
whenever you want. 


Thesis Experiment 
- Phase 1: Pre-Paperwork - 


Before we go more into detail about the 
experiment please fill out the required forms. 

From the papers provided to you, please read 
and sign the following forms now (see also the 
next slide): 

• NPS Participant Consent Form & Minimai 
Risk Statement 

• NPS Privacy Act Statement 


If you have any questions, do not hesitate to ask. s 
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When you are done, please continue with the presentation. 

e 



Thesis Experiment 
Phase 1: Pre-Papenworlt - 


From the additionai papers provided to you, 
piease read and sign on/y the initiai 
Questionnaire and ieave the fieid Participant iD 
biank. 


When you are done, please continue with the presentation. ^ 



Thesis Experiment 

V 

- Phase 2: introduction - 


Problem statement 

In manned as well as in unmanned aviation the 
pilot/ operator is sometimes faced with an 
unusuai aircraft attitude. Often, his task is to 
recover from this unusuai attitude. 

in reiation to the speed and/ or altitude it is very 
important to recover quickiy and accurateiy. 

Depending on your mission and current task, 
quick recovery may be important. 


s 
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Thesis Experiment 
Phase 2: introduction - 



What is unusual attitude? 

... “an unusual attitude is defined in the industry 
as one where pitch unintentionally exceeds 25 
degrees nose up or 10 degrees nose down, bank 
angle unintentionally exceeds 45 degrees, and 
the airspeed is inappropriate for the conditions.” 


For this experiment we will focus on pitch and 
bank angle. We will not monitor the airspeed, 
since this is not the main interest. 


9 



Thesis Experiment 
' Phase 2: Introduction 


General aviation information 


The basic flight maneuvers are displayed below 
For this experiment Roll (Bank) Angle and Pitch 
are the important ones. 


RN 


Thesis Experiment 
- Phase 2: Introduction - 


Instruments - Attitude Indicator 

Since the beginning, pilots have depended 
heavily on their instruments. One of the most 
important, and also one of the oldest 
instruments, is the attitude indicator which 
provides information to the pilot about his 
current situation in space. 

The Unmanned Aerial Vehicle (UAV) Predator, 
which is of interest in this experiment, displays 
the information of pitch and angle of bank in a 
Heads-Up-Display (HUD) that you can see on the 
following slide. 

11 
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Thesis Experiment 

V 

- Phase 2: introduction - 



It provides an , meaning a view from 

the inside of the aircraft to the outside worid. 



If aircraft reaches an unusual attitude pitch and/ or roll will appear 
in yellow in the left bottom corner of the display 
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Thesis Experiment 

V 

- Phase 2: introduction - 


Instruments - New Prototype Design, the WEBER-Box 
A new prototype display called the WEBER-Box 
has recently been designed. 

As opposed to the traditional design, it provides 
an , meaning a view from the 

outside world into the aircraft. 




Thesis Experiment 

V 

- Phase 2: introduction - 


Scale of the new Prototype Design 


Information 
pitch 

bank anyle 

16 




, Thesis Experiment 

- Phase 2: introduction - 


IT aircraft reaches an unusual attitude the celor of 
pitch/ bank angle will switch to yellow 


bank angle' 
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Thesis Experiment 


■ Phase 2: Training - 


jj. 

, iVi , 

.d_ * 

^ — ”• 



XI 




.«71 


Is this a usual/unusual attitude? 
, since level flight. 


Thesis Experiment 
- Phase 2: Training - 


' ' 




t —.r 




XI »•>“ 




Is this a usual/unusual attitude? 

, since pitch > 25° nose up. 


Thesis Experiment 
- Phase 2: Training - 


r: 

t 

i 

L 


Is this a usual/ unusual attitude? 

, since pitch > 10° nose down. 
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Thesis Experiment 
■ Phase 2: Training - 

1 -r- , 1 1 



, «*> , 

=4 

/ 1 - 


i — 



- 

!- 


f 1 

' ^ 


Is this a usual/ unusual attitude? 

.since pitch <25° nose up and 

bank angie<45°. 

24 


Thesis Experiment 
- Phase 2: Training - 



}.ou 


Is this a usual or unusual attitude? 

, since bank angle < 45°. 

25 


Thesis Experiment 
- Phase 2: Training - 



Is this an usuai/ unusuai attitude? 
, since bank angle > 45°. 


26 
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Thesis Experiment 


■ Phase 2: Training - 



14 ^ 

/t 


Is this a usual or unusual attitude? 

, since pitch > 25° nose up. 

27 


Thesis Experiment 
- Phase 2: Training - 



Is this a usual or unusual attitude? 

, since bank angle < 45°. 

28 


Thesis Experiment 
■ Phase 2: Training - 



Is this an usuai/ unusuai attitude? 


, since pitch <25° nose up and 

bank angle < 45°. 

29 
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Thesis Experiment 


■ Phase 2: Training - 



Is this an usual/ unusual attitude? 

, since level flight. 

30 



Thesis Experiment 
- Phase 2: Training - 


1 / 

/ 

1 



Is this an usual/ unusual attitude? 

, since bank angle > 45°. 

31 

V 

Thesis Experiment 
■ Phase 2: Training - 





1 ■ ■ 

/— 

i__ 



Is this an usual/ unusual attitude? 

, since pitch > 10° nose down. 

32 
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Thesis Experiment 


• Phase 3: Experiment - 


The Task 

You are currently assigned as a pilot of the UAV 
MQ-1 Predator. 



V 

Thesis Experiment 
- Phase 3: Experiment ■ 

Features of the Predator “For vour information” 

Primary Function: 

Armed reconnaissance, airborne surveillance, target 
acquisition 

Contractor: 

General Atomics Aeronautical Systems Incorporated 

Power Plant: 

Rotax 914 four cylinder engine with 101 horsepower 

Length: 

27 feet (8.22 meters) 

Height: 

6.9 feet (2.1 meters) 

Weight: 

1,130 pounds (512 kg) empty, maximum takeoff weight 
2,250 pounds (1,020 kg) 

Wingspan: 

48.7 feet (14.8 meters) 

Speed: 

Cruise speed around 84 mph (70 knots), up to 135 mph 

Flange: 

up to 400 nautical miles (454 miles) 

Ceiling: 

up to 25,000 feet (7,620 meters) 

Fuel Capacity: 

665 pounds (100 gallons) 

Payload: 

450 pounds (204 kg) 

Armament: 

two laser-guided AGM-114 Heilfire anti-tank missiles 34 


Thesis Experiment 

V 

- Phase 3: Experiment - 


General information 

The MQ-1 Predator is a medium-altitude, long-endurance, 
remotely piloted aircraft. It’s primary mission Is 
interdiction and conducting armed reconnaissance 
against critical, perishable targets. When It is not actively 
pursuing Its primary mission, it acts as the Joint Forces 
Air Component Commander-owned theater asset for 
reconnaissance, surveillance and target acquisition in 
support of the Joint Forces Commander. 

The basic crew Is one pilot and two sensor operators. 

They fly the aircraft from inside the ground control station 
via a line-of-sight data link or a satellite data link for 
beyond line-of-sight flight. 

35 
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Thesis Experiment 
- Phase 3: Experiment - 



General information 

The aircraft is equipped with a coior camera in the nose 
(generaily used by the pilot for flight control), a day 
variable-aperture TV camera, a variable-aperture infrared 
(IR) camera (for low lightfnight), and a synthetic aperture 
radar (SAR) for looking through smoke, clouds or haze. 
The cameras produce full motion video while the SAR 
produces still-frame radar images. 



Thesis Experiment 


• Phase 3: Experiment • 


The Task 


Your duty station is the 
Ground Control Station, 
located in Monterey, at the 
Naval Postgraduate School 

To gain a better 
understanding about the 
work of an operator of a 
Predator, please vi ew thi s 
video by pressing ^*8 



Thesis Experiment 
• Phase 3: Experiment - 


Experiment Goals 

The goal of this experiment is to test different flight 
instruments (Traditional HUD and new prototype design 
WEBER-Box) with different characteristics (color and 
non-color) for supporting judgment of aircraft attitude. 


Overall Experiment Schedule 

You will conduct four sets of trials. The order in which 
you will experience the four sets is assigned randomly. 
The only difference between the four sets is which set of 
flight instruments is used and if it is color-coded or not. 
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Thesis Experiment 
- Phase 3: Experiment - 
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APPENDIX E. PARTICIPANT CONSENT FORM & MINIMAL 

RISK STATEMENT 


Naval Postgraduate School 

Participant Consent Form & 
Minimal Risk Consent Statement 


Introduction, You are invited to participate in a study entitled Pictorial Display Design 
to enhance Spatial Awareness of Operators in Unmanned Aviation being conducted by 
the Naval Postgraduate School MOVES Institute. 

Procedures. If I agree to participate in this study, I understand that I will be provided with 
an explanation of the purposes of the research, a description of the procedures to be used, 
identification of any experimental procedures, and the expected duration of my 
participation. 

Synopsis : There will be two sessions: (1) a 30 minute test phase with traditional 
flight instruments and (2) a 30 minute test phase with the new prototype display design, 
during which you will be expected to accomplish a number of tasks related to orientation 
during flight operations. 

Risks and Benefits. I understand that this project does not involve greater than minimal 
risk and involves no known reasonably foreseeable risks or hazards greater than those 
encountered in everyday life. I understand that there is a very small chance that susceptible 
participants could experience symptoms of simulator sickness such as “queasy” stomach. I 
agree that, if I do experience such symptoms, I will remain in the testing area until they 
have subsided. I have also been informed of any benefits to myself or to others that may 
reasonably be expected as a result of this research. 

Compensation, I understand that no tangible reward will be given. I understand that a 
copy of the research results will be available at the conclusion of the experiment. 

Confidentiality & Privacy Act. I understand that all records of this study will be kept 
confidential and that my privacy will be safeguarded. No information will be publicly 
accessible which could identify me as a participant, and I will be identified only as a code 
number on all research forms. I understand that records of my participation will be 
maintained by NPS for five years, after which they will be destroyed. 

Voluntary Nature of the Study, I understand that my participation is strictly voluntary, 
and if I agree to participate, I am free to withdraw at any time without prejudice. 

Points of Contact, I understand that if I have any questions or comments regarding this 
project upon the completion of my participation, I should contact the Principal Investigator, 
Dr. Michael E. McCauley, 656-2191, memccaul@nps.edu. Any medical questions should 
be addressed to ETC Eric Morgan, MC, USA, (CO, POM Medical Clinic), (831) 242-7550, 
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eric ■ morgan @ nw.amedd.army.mil . Any other questions or concerns may be addressed to 
the IRB Chair, LT Brent Olde, 656-3807, baolde@nps.edu. 

Statement of Consent, I have read and understand the above information. I have asked 
all questions and have had my questions answered. I agree to participate in this study. I 
will be provided with a copy of this form for my records. 


Participant’s Signature 


Date 


Researcher’s Signature 


Date 
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APPENDIX F. PRIVACY ACT STATEMENT 


NAVAL POSTGRADUATE SCHOOL, MONTEREY, CA 93943 
PRIVACY ACT STATEMENT 


1. Purpose: Spatial cognition data will be collected to enhance knowledge, and to 
develop tests, procedures, and equipment to improve the development of Virtual 
Environments. 

2. Use: Spatial cognition data will be used for statistical analysis by the Departments of 
the Navy and Defense, and other U.S. Government agencies, provided this use is 
compatible with the purpose for which the information was collected. Use of the 
information may be granted to legitimate non-government agencies or individuals by 
the Naval Postgraduate School in accordance with the provisions of the Freedom of 
Information Act. 

3. Disclosure/Confidentiality: 

a. I have been assured that my privacy will be safeguarded. I will be assigned a 
control or code number which thereafter will be the only identifying entry on 
any of the research records. The Principal Investigator will maintain the cross- 
reference between name and control number. It will be decoded only when 
beneficial to me or if some circumstances, which is not apparent at this time, 
would make it clear that decoding would enhance the value of the research data. 
In all cases, the provisions of the Privacy Act Statement will be honored. 

b. I understand that a record of the information contained in this Consent Statement 
or derived from the experiment described herein will be retained permanently at 
the Naval Postgraduate School or by higher authority. I voluntarily agree to its 
disclosure to agencies or individuals indicated in paragraph 3 and I have been 
informed that failure to agree to such disclosure may negate the purpose for 
which the experiment was conducted. 

c. I also understand that disclosure of the requested information, including my 
Social Security Number, is voluntary. 


Name, Grade/Rank (if applicable) 
[Please print] 


Signature of Volunteer 
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APPENDIX G. INITIAL QUESTIONNAIRE 


Date: 


Initial Questionnaire 


Age 


Gender 

Male CH Female CH 


Are you experienced in aviation in any 
regard (incl, playing Flight Simulation 
Games)? 


YES □ 


NO □ 


If you answered the last question with 
YES, please proceed HERE._ 


Are you an aviator/pilot? 

YES □ NO □ 

Are you familiar with the basic set of 
flight instruments? 

YES □ NO □ 

Are you familiar with the basic aircraft 
flight controls? 

YES □ NO □ 

Did you operate any of these aircraft?: 
Fixed-Wing (Jet/Propeller) 

Rotor-Wing Aircraft 

UAV 

Fixed CH Rotor CH 

Jet CH Propeller CH 

UAV □ 

How many flight hours do you have? 

Hours 

How many hours in a flight simulator do 
you have? 

Hours 

How long ago was your last flight? 

Years Month(s) 

How long ago was your last use of a 
flight simulator? 

Years Month(s) 

Are you trained in procedures of 
recovery from unusual attitude? 

YES □ NO □ 
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Leave blank! 


^^^articipanO^ 


If you are an active/former aviator/pilot, 
please proceed HERE:_ 


Have you ever experienced any kind of 
Spatial Disorientation? 

YES □ NO □ 

More than once? 

YES □ NO □ 

Briefly describe the situation/s 



PLEASE STOP HERE! 
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APPENDIX H. POST-EXPERIMENT QUESTIONNAIRE 


Post-Experiment Questionnaire 




Question 


Answer-Opi 

tions 


In general, how do you judge the 
difficulty of evaluating the current 
attitude of the aircraft? 

Very 

easy 

Somewhat 

easy 

Border 

line 

Somewhat 

complicated 

Very difficult 

In general, how do you judge difficulty 
of using the traditional flight instruments 
for this task? 

Very 

easy 

Somewhat 

easy 

Border 

line 

Somewhat 

complicated 

Very difficult 

In general, how do you judge the 
difficulty of using the WEBER-Box for 
this task? 

Very 

easy 

Somewhat 

easy 

Border 

line 

Somewhat 

complicated 

Very difficult 

In general, how do you judge the 
benefits of color coding of an unusual 
attitude? 

Very 

good 

Somewhat 

good 

Border 

line 

Somewhat 

poor 

Very poor 

Did you feel better aware of the spatial 
orientation of the airplane with the 
WEBER-Box in comparison with the 
traditional flight instruments, when both 
are without color coding of an unusual 
attitude? 

Much less 

Somewhat 

less 

Border 

line 

Somewhat more 

Much more 

Did you feel better aware of the spatial 
orientation of the airplane with the 
WEBER-Box in comparison with the 
traditional flight instruments, when both 
are with color coding of an unusual 
attitude? 

Much less 

Somewhat 

less 

Border 

line 

Somewhat more 

Much more 

Did having the WEBER-Box increase or 
decrease your monitoring demands in 
comparison with the traditional flight 
instruments? 

Greatly 

decreased 

Somewhat 

decreased 

Unaffected 

Somewhat 

increased 

Greatly 

increased 
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APPENDIX I. COMMENTS/ SUGGESTIONS/ OPINIONS 



In General: 


Experiment Tasks: 


Traditional Instruments: 


WEBER-Box: 
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APPENDIX J. ORDER OF TRIALS 


Order of Trials 


Project Study 02 


Participant 

Order 

1. 

SOI 

NP 

A 

B 

C 

D 

2. 

SIO 

P 

B 

C 

D 

A 

3. 


NP 

C 

D 

A 

B 


Experiment 


Participant 

Order 

1. 

201 

NP 

A 

B 

C 

D 

2. 

101 

P 

B 

C 

D 

A 

3. 

202 

NP 

C 

D 

A 

B 

4. 

001 

P 

D 

A 

B 

C 

5. 

203 

NP 

A 

B 

C 

D 

6. 

002 

P 

B 

C 

D 

A 

7. 

003 

P 

C 

D 

A 

B 

8. 

004 

P 

D 

A 

B 

C 

9. 

005 

P 

A 

B 

C 

D 

10. 

204 

NP 

B 

C 

D 

A 

11. 

205 

NP 

C 

D 

A 

B 

12. 

006 

P 

D 

A 

B 

C 

13. 

206 

NP 

A 

B 

C 

D 

14. 

207 

NP 

B 

C 

D 

A 

15. 

007 

P 

C 

D 

A 

B 

16. 

208 

NP 

D 

A 

B 

C 

17. 

209 

NP 

A 

B 

C 

D 

18. 

008 

P 

B 

C 

D 

A 

19. 

102 

P 

C 

D 

A 

B 

20. 

210 

NP 

D 

A 

B 

C 


Legend: A 

B 
C 
D 

P 

NP 


Traditional Design without color 
Prototype Design without color 
Traditional Design with color 
Prototype Design with color 

Pilot 

Non-Pilot 
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APPENDIX K: EVALUATION FORM 




Start with Traditional Design without color : A □ 


Trails 

Attitude 

correct 

incorr. 

1 - test trial 

A 

usual 

pitch 15° nose up 




2 - test trial 

B 

usual 

bank angle 25° left 




3 - test trail 

C 

unusual 

bank angle 60° right 




4 - test trail 

D 

unusual 

pitch 15° nose down 




5 - test trail 

E 

unusual 

pitch 35° nose up & 
bank angle 55° left 




6 

M 

unusual 

pitch 30° nose up 




7 

H 

usual 

bank angle 35° left 




8 

J 

usual 

pitch 10° nose up 
bank angle 20° right 

& 



9 

0 

unusual 

bank angle 65° left 




10 

G 

usual 

pitch 05° nose down 




11 

Q 

unusual 

pitch 05° nose down 
bank angle 55° left 

& 



12 

N 

unusual 

pitch 20° nose down 




13 

F 

usual 

pitch 20° nose up 




14 

P 

unusual 

bank angle 55° right 




15 

R 

unusual 

pitch 25° nose down 
bank angle 35° left 

& 



16 

K 

usual 

pitch 05° nose down 
bank angle 35° left 

& 



17 

I 

usual 

bank angle 25° right 




18 

S 

unusual 

pitch 40° nose up 
bank angle 60° right 

& 



19 

L 

usual 

pitch 5° nose down 
bank angle 25° right 

& 


























Start with Traditional Design with color: C □ 


Trails 

Attitude 

correct 

incorr. 

Time 

1 - test trial 

A 

usual 

pitch 15° nose up 





2 - test trial 

B 

usual 

bank angle 25° left 





3 - test trail 

C 

unusual 

bank angle 60° right 





4 - test trail 

D 

unusual 

pitch 15° nose down 





5 - test trail 

E 

unusual 

pitch 35° nose up & 
bank angle 55° left 





6 

P 

unusual 

bank angle 55° right 





7 

M 

unusual 

pitch 30° nose up 





8 

K 

usual 

pitch 05° nose down 
bank angle 35° left 

& 




9 

N 

unusual 

pitch 20° nose down 





10 

I 

usual 

bank angle 25° right 





11 

J 

usual 

pitch 10° nose up 
bank angle 20° right 

& 




12 

0 

unusual 

bank angle 65° left 





13 

s 

unusual 

pitch 40° nose up 
bank angle 60° right 

& 




14 

G 

usual 

pitch 05° nose down 





15 

R 

unusual 

pitch 25° nose down 
bank angle 35° left 

& 




16 

H 

usual 

bank angle 35° left 





17 

Q 

unusual 

pitch 05° nose down 
bank angle 55° left 

& 




18 

L 

usual 

pitch 5° nose down 
bank angle 25° right 

& 




19 

F 

usual 

pitch 20° nose up 
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Start with WEBER-Box without color: B □ 


Trails 

Attitude 

correct 

incorr. 

Time 

1 - test trial 

A 

usual 

pitch 15° nose up 





2 - test trial 

B 

usual 

bank angle 25° left 





3 - test trail 

C 

unusual 

bank angle 60° right 





4 - test trail 

D 

unusual 

pitch 15° nose down 





5 - test trail 

E 

unusual 

pitch 35° nose up & 
bank angle 55° left 





6 

I 

usual 

bank angle 25° right 





7 

G 

usual 

pitch 05° nose down 





8 

0 

unusual 

bank angle 65° left 





9 

P 

unusual 

bank angle 55° right 





10 

N 

unusual 

pitch 20° nose down 





11 

H 

usual 

bank angle 35° left 





12 

K 

usual 

pitch 05° nose down 
bank angle 35° left 

& 




13 

M 

unusual 

pitch 30° nose up 





14 

F 

usual 

pitch 20° nose up 





15 

Q 

unusual 

pitch 05° nose down 
bank angle 55° left 

& 




16 

J 

usual 

pitch 10° nose up 
bank angle 20° right 

& 




17 

R 

unusual 

pitch 25° nose down 
bank angle 35° left 

& 




18 

L 

usual 

pitch 5° nose down 
bank angle 25° right 

& 




19 

S 

unusual 

pitch 40° nose up 
bank angle 60° right 

& 
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Start with WEBER-Box with color: D □ 


Trails 

Attitude 

correct 

incorr. 

Time 

1 - test trial 

A 

usual 

pitch 15° nose up 





2 - test trial 

B 

usual 

bank angle 25° left 





3 - test trail 

C 

unusual 

bank angle 60° right 





4 - test trail 

D 

unusual 

pitch 15° nose down 





5 - test trail 

E 

unusual 

pitch 35° nose up & 
bank angle 55° left 





6 

Q 

unusual 

pitch 05° nose down 
bank angle 55° left 

& 




7 

F 

usual 

pitch 20° nose up 





8 

K 

usual 

pitch 05° nose down 
bank angle 35° left 

& 




9 

0 

unusual 

bank angle 65° left 





10 

N 

unusual 

pitch 20° nose down 





11 

R 

unusual 

pitch 25° nose down 
bank angle 35° left 

& 




12 

L 

usual 

pitch 5° nose down 
bank angle 25° right 

& 




13 

I 

usual 

bank angle 25° right 





14 

M 

unusual 

pitch 30° nose up 





15 

J 

usual 

pitch 10° nose up 
bank angle 20° right 

& 




16 

G 

usual 

pitch 05° nose down 





17 

P 

unusual 

bank angle 55° right 





18 

S 

unusual 

pitch 40° nose up 
bank angle 60° right 

& 




19 

H 

usual 

bank angle 35° left 
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Legend: 


Index 

Attitude 

Pitch 

Bank angle 


usual 

pitch 15° nose up 


B 

usual 


bank angle 25° left 

C 

unusual 


bank angle 60° right 

D 

unusual 

pitch 15° nose down 


E 

unusual 

pitch 35° nose up 

& 

bank angle 55° left 

F 

usual 

pitch 20° nose up 


G 

usual 

pitch 05° nose down 


H 

usual 


bank angle 35° left 

I 

usual 


bank angle 25° right 

J 

usual 

pitch 10° nose up 

& 

bank angle 20° right 

K 

usual 

pitch 5° nose down 

& 

bank angle 35° left 

L 

usual 

pitch 5° nose down 

& 

bank angle 25° right 

M 

unusual 

pitch 30° nose up 


N 

unusual 

pitch 20° nose down 


O 

unusual 


bank angle 65° left 

P 

unusual 


bank angle 55° right 

Q 

unusual 

pitch 5° nose down 

& 

bank angle 55° left 

R 

unusual 

pitch 25° nose down 

& 

bank angle 35° left 

S 

unusual 

pitch 40° nose up 

& 

bank angle 60° right 
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